Markdown documentation for the latest release
Developer Guide - Table of Contents
Start by installing requisites and the rapydo-controller (as for the User guide)
Make sure you meet the pre-requisites on your machine:
pip)install the rapydo controller
Install the latest version: pip3 install --upgrade rapydo
Your project could require a different version, in this case you will be able to install the right version once configured your project
Create and enter a new directory
Create a new project by using
rapydo create name --auth YOUR_AUTH_SERVICE --frontend YOUR_FRONTEND ...other-options
Now you can edit your projects/$PROJECT_NAME/project_configuration.yaml to customize project title, description, enabled services and so on. Please note that you can enable services, ad other options, by setting up the rapydo create command. Use rapydo create --help to get the list of available options.
You can list active services with rapydo list services
Active services are a combination of services enabled by configuring the ACTIVATE_SERVICENAME variable in projects/$PROJECT_NAME/project_configuration.yaml or .projectrc and services automatically activated due to dependency rules. At creation time you can enable new services by adding the --service flag to the rapydo create command
By activating a service the backend server will establish a connection by using one of the configured extension:
If you want to activate the service but NOT the connection from the backend you can add a SERVICENAME_ENABLE_CONNECTOR=False variable in projects/$PROJECT_NAME/project_configuration.yaml or .projectrc
If you want to activate the connection from the backend but NOT a container for the service, you can configure the SERVICENAME_HOST with an external host.
To summarize:
Service container enabled + Backend connection enable -> ACTIVATE_SERVICENAME
Service container enabled + Backend connection disabled -> ACTIVATE_SERVICENAME && SERVICENAME_ENABLE_CONNECTOR
Service container disabled + Backend connection enable -> SERVICENAME_HOST
Service container disabled + Backend connection disabled -> nothing
RAPyDo is designed to be easily extended with new services in addition to those already provided. For instance, let’s suppose we want to add Apache NiFi. We have to add the new service in {custom}/confs/common.yml and activate it from the {custom}/project_configuration.yml
common.yml:
services:
...
nifi:
image: apache/nifi
ports:
- 8070:8080
environment:
ACTIVATE: ${ACTIVATE_NIFI}
and activate it in project_configuration.yml
variables:
env:
...
ACTIVATE_NIFI: 1
and rapydo start will do the rest
RAPyDo supports several backend database to store authentication information: relation databases via sqlalchemy (postgres, mysql, mariadb), neo4j, mongodb. You can set the preferred database by setting the AUTH_SERVICE variable. You can also disable the authentication layer at all by setting AUTH_SERVICE=NO_AUTHENTICATION.
A number of flags allow to enhance the security layer, included enabling 2-Factor authentication based on TOTP.
Here a list of variabiles that can be configured in project_configuration.yaml or in .projectrc:
RAPyDO is Object Oriented (based to the Flask and flask-apispec packages): each endpoint is mapped to a class and configured automatically. You can simply create a new python file in projects/YOUR_PROJECT/backend/endpoinits folder to define your class-endpoint and it will be automatically added to the project. You can add new endpoints by using the templating command: rapydo add endpoint endpoint-name
A REST class extends EndpointResource from restapi.rest.definition, defines methods get, post, put, patch or deleted decorated by the @decorators.endpoint decorator used to describe path, description and responses. This decorator is mostly based on apispec. Input and output can be specified respectively by the @decorators.use_kwargs(mostly based on webargs) and @decorators.marshal_with (mostly based on Marshmallow) decorators.
Here an example of a simple REST class:
from restapi import decorators
from restapi.rest.definition import EndpointResource, Response
class MyFirstRESTClass(EndpointResource):
# @decorators.auth.require()
# @decorators.use_kwargs(MyWebargsInputSchema)
# @decorators.marshal_with(MyMarshalOutputSchema)
@decorators.endpoint(
path="/api/myendpoint",
summary="My first endpoint",
description="This is my first endpoint, it simply reply with a Hello World",
responses={
200: "Hello World is replied",
},
)
def get(self) -> Response:
self.response("Hello World")
Helper endpoints are provided out of the box:
/api/status/api/specs/auth/login/auth/logout/auth/tokens/auth/profile/auth/status/api/admin/users/api/admin/sessions/api/admin/stats/api/groups/usersBy default the backend embeds a wrapper of loguru that can be used by importing from the restapi.utilities package:
from restapi.utilities.logs import log
...
log.debug("My message")
log.info("My message")
log.warning("My message")
log.error("My message")
log.critical("My message")
Logs visibility is controlled by a global variable (LOG_LEVEL) that can be modified at project_configuration and .projectrc level (default value is DEBUG).
By default all the logs with severity equal or greater then the warning level are saved to the data/logs/backend-server.log file. This file is automatically rotated every week and retained for 6 months (controller by the LOG_RETENTION variable).
the log.exception function is not printed in standard output but only stored on the backend-server.log with a detailed error stack for debugging purpose.
As an additional logging system, from every endpoint it possible to store information on the data/logs/security-events.log. As for the general log, also the security events log is automatically rotated every week and retained for 6 months (controller by the LOG_RETENTION variable).
To store onto the security events logs the endpoints the use the log_event utility:
myobj = db.myModel(kwargs).save()
self.log_event(self.events.create, myobj, kwargs)
Supported events type (first parameter) are:
The second parameter is the event target (usually the object created/modified/delete) while the third is the payload used for the event. Both the parameters are optional.
The log is automatically extended with date, IP, user (in case of authenticated endpoint), object type and object id/uuid
Celery tasks can be easily activated to be able to launch and control from any endpoint asynchronous tasks.
Tasks can be monitored directly from endpoints or from the flower UI.
Tasks can save progress, return status, send emails.
py.tests is supported by default.
A unit test is a class in a separated tests folder, where you extend the existing base class from where you inherit methods to authenticate and handle tokens.
Angular is already integrated as base framework for the frontend part.
The base authentication (profile, change password, reset password, session lists, JWT tokens) endpoints are already tested inside the base TS code.
In debug mode the framework is dynamically compiled and provided by ng serve while in production the static dist is built at startup time and served by a nginx reverse proxy.
If your project is based on version A and you want to upgrade it to version B:
rapydo install)rapydo/base-image:A to rapydo/base-image:BYour project is now upgraded to version B.