irods-capability-automated-ingest 0.5.0

Description:

irodscapabilityautomatedingest 0.5.0

iRODS Automated Ingest Framework
The automated ingest framework gives iRODS an enterprise solution that solves two major use cases: getting existing data under management and ingesting incoming data hitting a landing zone.
Based on the Python iRODS Client and Celery, this framework can scale up to match the demands of data coming off instruments, satellites, or parallel filesystems.
The example diagrams below show a filesystem scanner and a landing zone.


Usage options
Redis options



option
effect
default




redis_host
Domain or IP address of Redis host
localhost


redis_port
Port number for Redis
6379


redis_db
Redis DB number to use for ingest
0



S3 options
To scan S3 bucket, minimally requires --s3_keypair and source path of the form /bucket_name/path/to/root/folder.



option
effect
default




s3_keypair
path to S3 keypair file
None


s3_endpoint_domain
S3 endpoint domain
s3.amazonaws.com


s3_region_name
S3 region name
us-east-1


s3_proxy_url
URL to proxy for S3 access
None


s3_insecure_connection
Do not use SSL when connecting to S3 endpoint
False



Logging/Profiling options



option
effect
default




log_filename
Path to output file for logs
None


log_level
Minimum level of message to log
None


log_interval
Time interval with which to rollover ingest log file
None


log_when
Type/units of log_interval (see TimedRotatingFileHandler)
None



--profile allows you to use vis to visualize a profile of Celery workers over time of ingest job.



option
effect
default




profile_filename
Specify name of profile filename (JSON output)
None


profile_level
Minimum level of message to log for profiling
None


profile_interval
Time interval with which to rollover ingest profile file
None


profile_when
Type/units of profile_interval (see TimedRotatingFileHandler)
None



Ingest start options
These options are used at the "start" of an ingest job.



option
effect
default




job_name
Reference name for ingest job
a generated uuid


interval
Restart interval (in seconds). If absent, will only sync once.
None


file_queue
Name for the file queue.
file


path_queue
Name for the path queue.
path


restart_queue
Name for the restart queue.
restart


event_handler
Path to event handler file
None (see "event_handler methods" below)


synchronous
Block until sync job is completed
False


progress
Show progress bar and task counts (must have --synchronous flag)
False


ignore_cache
Ignore last sync time in cache - like starting a new sync
False



Optimization options



option
effect
default




exclude_file_type
types of files to exclude: regular, directory, character, block, socket, pipe, link
None


exclude_file_name
a list of space-separated python regular expressions defining the file names to exclude such as "(\S+)exclude" "(\S+).hidden"
None


exclude_directory_name
a list of space-separated python regular expressions defining the directory names to exclude such as "(\S+)exclude" "(\S+).hidden"
None


files_per_task
Number of paths to process in a given task on the queue
50


initial_ingest
Use this flag on initial ingest to avoid check for data object paths already in iRODS
False


irods_idle_disconnect_seconds
Seconds to hold open iRODS connection while idle
60



available --event_handler methods



method
effect
default




pre_data_obj_create
user-defined python
none


post_data_obj_create
user-defined python
none


pre_data_obj_modify
user-defined python
none


post_data_obj_modify
user-defined python
none


pre_coll_create
user-defined python
none


post_coll_create
user-defined python
none


pre_coll_modify
user-defined python
none


post_coll_modify
user-defined python
none


character_map
user-defined python
none


as_user
takes action as this iRODS user
authenticated user


target_path
set mount path on the irods server which can be different from client mount path
client mount path


to_resource
defines target resource request of operation
as provided by client environment


operation
defines the mode of operation
Operation.REGISTER_SYNC


max_retries
defines max number of retries on failure
0


timeout
defines seconds until job times out
3600


delay
defines seconds between retries
0



Event handlers can use logger to write logs. See structlog for available logging methods and signatures.
Operation mode



operation
new files
updated files




Operation.REGISTER_SYNC (default)
registers in catalog
updates size in catalog


Operation.REGISTER_AS_REPLICA_SYNC
registers first or additional replica
updates size in catalog


Operation.PUT
copies file to target vault, and registers in catalog
no action


Operation.PUT_SYNC
copies file to target vault, and registers in catalog
copies entire file again, and updates catalog


Operation.PUT_APPEND
copies file to target vault, and registers in catalog
copies only appended part of file, and updates catalog


Operation.NO_OP
no action
no action



--event_handler usage examples can be found in the examples directory.
Character Mapping option
If an application should require that iRODS logical paths produced by the ingest process exclude subsets of the
range of possible Unicode characters, we can add a character_map method that returns a dict object. For example:
class event_handler(Core):
@staticmethod
def character_map():
return {
re.compile('[^a-zA-Z0-9]'):'_'
}
# ...

The returned dictionary, in this case, indicates that the ingest process should replace all non-alphanumeric (as
well as non-ASCII) characters with an underscore wherever they may occur in an otherwise normally generated logical path.
The substition process also applies to the intermediate (ie collection name) elements in a logical path, and a suffix is
appended to affected path elements to avoid potential collisions with other remapped object names.
Each key of the returned dictionary indicates a character or set of characters needing substitution.
Possible key types include:

character

# substitute backslashes with underscores
'\\': '_'


tuple of characters

# any character of the tuple is replaced by a Unicode small script x
('\\','#','-'): '\u2093'


regular expression

# any character outside of range 0-256 becomes an underscore
re.compile('[\u0100-\U0010ffff]'): '_'


callable accepting a character argument and returning a boolean

# ASCII codes above 'z' become ':'
(lambda c: ord(c) in range(ord('z')+1,128)): ':'

In the event that the order-of-substitution is significant, the method may instead return a list of key-value tuples.
UnicodeEncodeError
Any file whose path in the filesystem whose ingest results in a UnicodeEncodeError exception being raised (e.g. by the
inclusion of an unencodable UTF8 sequence) will be automatically renamed using a base-64 sequence to represent the original,
unmodified vault path.
Additionally, data objects that have had their names remapped, whether pro forma or via a UnicodeEncodeError, will be
annotated with an AVU of the form
Attribute: "irods::automated_ingest::" + ANNOTATION_REASON
Value: A PREFIX plus the base64-converted "bad filepath"
Units: "python3.base64.b64encode(full_path_of_source_file)"
Where :

ANNOTATION_REASON is either "UnicodeEncodeError" or "character_map" depending on why the remapping occurred.
PREFIX is either "irods_UnicodeEncodeError_" or blank(""), again depending on the re-mapping cause.

Note that the UnicodeEncodeError type of remapping is unconditional, whereas the character remapping is contingent on
an event handler's character_map method being defined. Also, if a UnicodeEncodeError-style ingest is performed on a
given object, this precludes character mapping being done for the object.
Manual Deployment
Configure python-irodsclient environment
python-irodsclient (PRC) is used by the Automated Ingest tool to interact with iRODS. The configuration and client environment files used for a PRC application applies here as well.
If you are using PAM authentication, remember to use the Client Settings File.
iRODSSessions are instantiated using an iRODS client environment file. The client environment file used can be controlled with the IRODS_ENVIRONMENT_FILE environment variable. If no such environment variable is set, the file is expected to be found at ${HOME}/.irods/irods_environment.json. A secure connection can be made by making the appropriate configurations in the client environment file.
Starting Redis Server
Install Redis server: https://redis.io/docs/latest/get-started
Starting the Redis server with package installation:
redis-server

Or, dameonized:
sudo service redis-server start

sudo systemctl start redis

The Redis GitHub page also describes how to build and run Redis: https://github.com/redis/redis?tab=readme-ov-file#running-redis
The Redis documentation also recommends an additional step:

Make sure to set the Linux kernel overcommit memory setting to 1. Add vm.overcommit_memory = 1 to /etc/sysctl.conf and then reboot or run the command sysctl vm.overcommit_memory=1 for this to take effect immediately.

This allows the Linux kernel to overcommit virtual memory even if this exceeds the physical memory on the host machine. See kernel.org documentation for more information.
Note: If running in a distributed environment, make sure Redis server accepts connections by editing the bind line in /etc/redis/redis.conf or /etc/redis.conf.
Setting up virtual environment
You may need to upgrade pip:
pip install --upgrade pip

Install virtualenv:
pip install virtualenv

Create a virtualenv with python3:
virtualenv -p python3 rodssync

Activate virtual environment:
source rodssync/bin/activate

Install this package
pip install irods_capability_automated_ingest

Set up environment for Celery:
export CELERY_BROKER_URL=redis://<redis host>:<redis port>/<redis db> # e.g. redis://127.0.0.1:6379/0
export PYTHONPATH=`pwd`

Start celery worker(s):
celery -A irods_capability_automated_ingest.sync_task worker -l error -Q restart,path,file -c <num workers>

Note: Make sure queue names match those of the ingest job (default queue names shown here).
Using the sync script
Start sync job
python -m irods_capability_automated_ingest.irods_sync start <source dir> <destination collection>

List jobs
python -m irods_capability_automated_ingest.irods_sync list

Stop jobs
python -m irods_capability_automated_ingest.irods_sync stop <job name>

Watch jobs (same as using --progress)
python -m irods_capability_automated_ingest.irods_sync watch <job name>

Run tests
Note: The tests start and stop their own Celery workers, and they assume a clean Redis database.
python -m irods_capability_automated_ingest.test.test_irods_sync

See docker/ingest-test/README.md for how to run tests with Docker Compose.