Packages

ssl.py

Import:

labpack.authentication.ssl

Description:

a package of methods for managing ssl authentication

generate_keystore

Signature:
generate_keystore(key_alias, key_folder=”./”, root_cert=”“, truststore=”“, password=”“, organization=”“, organization_unit=”“, locality=”“, country=”“, key_size=2048, verbose=True, overwrite=False)

Description:
a function to generate a keystore and cert files for self-signed ssl authentication

Argument	Type	Required	Default	Description
key_alias	NoneType	Yes	None
key_folder	str		"./"
root_cert	str		""
truststore	str		""
password	str		""
organization	str		""
organization_unit	str		""
locality	str		""
country	str		""
key_size	int		2048
verbose	bool		True
overwrite	bool		False

drep.py

Import:

labpack.compilers.drep

Description:

dump

Signature:
dump(json_input, secret_key)

Description:

Argument	Type	Required	Default	Description
json_input	NoneType	Yes	None
secret_key	NoneType	Yes	None

load

Signature:
load(encrypted_data, secret_key)

Description:

Argument	Type	Required	Default	Description
encrypted_data	NoneType	Yes	None
secret_key	NoneType	Yes	None

encoding.py

Import:

labpack.compilers.encoding

Description:

a package for encoding/decoding record data from ext type

decode_data

Signature:
decode_data(file_name, byte_data, mimetype=”“, secret_key=”“)

Description:

Argument	Type	Required	Default	Description
file_name	NoneType	Yes	None
byte_data	NoneType	Yes	None
mimetype	str		""
secret_key	str		""

encode_data

Signature:
encode_data(file_name, python_object, mimetype=”“, secret_key=”“)

Description:

Argument	Type	Required	Default	Description
file_name	NoneType	Yes	None
python_object	NoneType	Yes	None
mimetype	str		""
secret_key	str		""

filters.py

Import:

labpack.compilers.filters

Description:

a package of methods to compile search filters

positional_filter

Signature:
positional_filter(positional_filters, title=”“)

Description:
a method to construct a conditional filter function to test positional arguments

Argument	Type	Required	Default	Description
positional_filters	dict	Yes	None	dictionary or list of dictionaries with query criteria
title	str		""	string with name of function to use instead

git.py

Import:

labpack.compilers.git

Description:

merge_diff

Signature:
merge_diff(target, source, output=”“)

Description:
a method to merge the non-conflicting diffs between two files

    method retrieves the results from `git diff --no-index target source`
    and adds to target the lines from additions found in source. diff
    results which would subtract lines from target are ignored.

    PLEASE NOTE:    method makes no check to ensure that target, source
                    or output are valid paths

Argument	Type	Required	Default	Description
target	str	Yes	""	string with path to file for target data
source	str	Yes	""	string with path to file for source data
output	str		""	[optional] string with path to file to save output of merge

json.py

Import:

labpack.compilers.json

Description:

a package of methods to merge two or more json documents preserving order

merge_json

Signature:
merge_json(*sources, output=”“)

Description:
method for merging two or more json files

this method walks the parse tree of json data to merge the fields
found in subsequent sources into the data structure of the initial source. 
any number of sources can be added to the source args, but only new fields
from subsequent sources will be added. to overwrite values in the initial
source instead, it suffices to simply reverse the order of the sources

PLEASE NOTE:    since there is no way to uniquely identify list items between
                two json documents, items are not added to existing lists.

PLEASE NOTE:    however, lists are transversed in order to evaluate keys of 
                nested dictionaries using the first item of any subsequent list
                as a model for the scope

PLEASE NOTE:    this method makes no checks to ensure the file path of the 
                sources exist nor the folder path to any output

Argument	Type	Required	Default	Description
*sources	NoneType	Yes	None
output	str		""	[optional] string with path to save the combined json data to file

walk_data

Signature:
walk_data(target, source)

Description:
method to recursively walk parse tree and merge source into target

Argument	Type	Required	Default	Description
target	NoneType	Yes	None
source	NoneType	Yes	None

objects.py

Import:

labpack.compilers.objects

Description:

retrieve_function

Signature:
retrieve_function(function_string, global_scope=None, root_path=”./”)

Description:

Argument	Type	Required	Default	Description
function_string	NoneType	Yes	None
global_scope	NoneType		None
root_path	str		"./"

yaml.py

Import:

labpack.compilers.yaml

Description:

a package of methods to merge two or more yaml documents preserving order & comments

get_comments_map

Signature:
get_comments_map(self, key, default=None)

Description:

Argument	Type	Required	Default	Description
self	object	Yes	None
key	NoneType	Yes	None
default	NoneType		None

get_comments_seq

Signature:
get_comments_seq(self, idx, default=None)

Description:

Argument	Type	Required	Default	Description
self	object	Yes	None
idx	NoneType	Yes	None
default	NoneType		None

merge_yaml

Signature:
merge_yaml(*sources, output=”“)

Description:
method for merging two or more yaml strings

this method walks the parse tree of yaml data to merge the fields
(and comments) found in subsequent sources into the data structure of the
initial sources. any number of sources can be added to the source args, but
only new fields and new comments from subsequent sources will be added. to
overwrite the values in the initial source, it suffices to simply reverse 
the order of the sources

PLEASE NOTE:    since there is no way to uniquely identify list items between
                two yaml documents, items are not added to existing lists.
                the overwrite rule also has no effect on items in lists

PLEASE NOTE:    however, lists are transversed in order to evaluate comments
                and keys of nested dictionaries using the first item of any
                subsequent list as a model for the scope

PLEASE NOTE:    the way that ruamel.yaml keeps track of multi-line comments
                can create odd results for comments which appear at the start
                or the end of lists and dictionaries when new fields and comments 
                are added. it is best to restrict comments to the start of lists
                and dictionaries.

PLEASE NOTE:    this method makes no checks to ensure the file path of the 
                sources exist nor the folder path to any output

Argument	Type	Required	Default	Description
*sources	NoneType	Yes	None
output	str		""	[optional] string with path to save the combined yaml data to file

merge_yaml_strings

Signature:
merge_yaml_strings(*sources, output=”“)

Description:
method for merging two or more yaml strings

this method walks the parse tree of yaml data to merge the fields
(and comments) found in subsequent sources into the data structure of the
initial sources. any number of sources can be added to the source args, but
only new fields and new comments from subsequent sources will be added. to
overwrite the values in the initial source, it suffices to simply reverse 
the order of the sources

PLEASE NOTE:    since there is no way to uniquely identify list items between
                two yaml documents, items are not added to existing lists.

PLEASE NOTE:    however, lists are transversed in order to evaluate comments
                and keys of nested dictionaries using the first item of any
                subsequent list as a model for the scope

PLEASE NOTE:    the way that ruamel.yaml keeps track of multi-line comments
                can create odd results for comments which appear at the start
                or the end of lists and dictionaries when new fields and comments 
                are added. it is best to restrict comments to the start of lists
                and dictionaries.

Argument	Type	Required	Default	Description
*sources	NoneType	Yes	None
output	str		""	[optional] string with type of output: '' [default], io

walk_data

Signature:
walk_data(target, source)

Description:
method to recursively walk parse tree and merge source into target

Argument	Type	Required	Default	Description
target	NoneType	Yes	None
source	NoneType	Yes	None

iso_3166.py

Import:

labpack.datasets.iso_3166

Description:

a package of methods for compiling information about ISO 3166 country codes

compile_list

Signature:
compile_list(csv_file=”datasets/iso_3166.csv”)

Description:

compile_map

Signature:
compile_map(key_column=”Alpha-3code”, csv_list=None)

Description:

Argument	Type	Required	Default	Description
key_column	str		"Alpha-3code"
csv_list	NoneType		None

update_csv

Signature:
update_csv(csv_url=”“)

Description:

iso_3166_2_US.py

Import:

labpack.datasets.iso_3166_2_US

Description:

a package of methods for compiling information about ISO 3166 2 US state codes

compile_list

Signature:
compile_list(csv_file=”datasets/iso_3166_2_US.csv”)

Description:

compile_map

Signature:
compile_map(key_column=”USPS”, csv_list=None)

Description:

Argument	Type	Required	Default	Description
key_column	str		"USPS"
csv_list	NoneType		None

update_csv

Signature:
update_csv(csv_url=”“)

Description:

cryptolab.py

Import:

labpack.encryption.cryptolab

Description:

decrypt

Signature:
decrypt(encrypted_data, secret_key)

Description:
uses cryptography module to decrypt byte data cipher: AES (128 bit block_size) hash: sha512 key size: 256 bit (first 32 bytes of secret key hash) vector size: 128 bit (next 16 bytes of secret key hash) padding: PKCS7 cipher mode: CBC backend: openssl 1.0.2a

Argument	Type	Required	Default	Description
encrypted_data	bytes	Yes	None	bytes with data to decrypt
secret_key	str	Yes	""	[optional] string used to decrypt data

encrypt

Signature:
encrypt(byte_data, secret_key=”“)

Description:
uses cryptography module to encrypt byte data cipher: AES (128 bit block_size) hash: sha512 key size: 256 bit (first 32 bytes of secret key hash) vector size: 128 bit (next 16 bytes of secret key hash) padding: PKCS7 cipher mode: CBC backend: openssl 1.0.2a

    NOTE:   if secret_key is left blank,
            method generates a 32 byte hexadecimal string

Argument	Type	Required	Default	Description
byte_data	bytes	Yes	None	bytes with data to encrypt
secret_key	str		""	[optional] string used to encrypt data

requests.py

Import:

labpack.handlers.requests

Description:

handle_requests

Signature:
handle_requests(request_object, uptime_url=”www.google.com”)

Description:

Argument	Type	Required	Default	Description
request_object	NoneType	Yes	None
uptime_url	str		"www.google.com"

requestsHandler

Signature:
requestsHandler(self, uptime_url=”www.google.com”, requests_handler=”handle_requests”, response_handler=None, verbose=False)

Description:
the initialization method for the requestsHandler class object

Argument	Type	Required	Default	Description
self	object	Yes	None
uptime_url	str		"www.google.com"	[optional] string with url to test availability of internet
requests_handler	function		"handle_requests"	[optional] callable method which accepts a Request object
response_handler	function		None	[optional] callable method which accepts a Response object
verbose	bool		False	boolean to enable print out of status

data.py

Import:

labpack.mapping.data

Description:

clean_data

Signature:
clean_data(input_value)

Description:
a function to transform a value into a json or yaml valid datatype

reconstruct_dict

Signature:
reconstruct_dict(dot_paths, values)

Description:
a method for reconstructing a dictionary from the values along dot paths

Argument	Type	Required	Default	Description
dot_paths	NoneType	Yes	None
values	NoneType	Yes	None

segment_path

Signature:
segment_path(dot_path)

Description:
a function to separate the path segments in a dot_path key

transform_data

Signature:
transform_data(function, input_data)

Description:
a function to apply a function to each value in a nested dictionary

Argument	Type	Required	Default	Description
function	function	Yes	None	callable function with a single input of any datatype
input_data	dict	Yes	None	dictionary or list with nested data to transform

walk_data

Signature:
walk_data(input_data)

Description:
a generator function for retrieving data in a nested dictionary

comparison.py

Import:

labpack.parsing.comparison

Description:

a package of methods to generate the differences between two data architectures

compare_records

Signature:
compare_records(new_record, old_record)

Description:
a method to generate the differences between two data architectures

Argument	Type	Required	Default	Description
new_record	list	Yes	None	set, list or dictionary with new details of an item
old_record	list	Yes	None	set, list or dictionary with old details of an item

conversion.py

Import:

labpack.parsing.conversion

Description:

a package of functions to convert data architecture formats

camelcase_to_lowercase

Signature:
camelcase_to_lowercase(camelcase_input, python_input=None)

Description:
a function to recursively convert data with camelcase key names into lowercase keys

Argument	Type	Required	Default	Description
camelcase_input	list	Yes	None	list or dictionary with camelcase keys
python_input	list		None	[optional] list or dictionary with default lowercase keys in output

lowercase_to_camelcase

Signature:
lowercase_to_camelcase(python_input, camelcase_input=None)

Description:
a function to recursively convert data with lowercase key names into camelcase keys

Argument	Type	Required	Default	Description
python_input	list	Yes	None	[optional] list or dictionary with default camelcase keys in output
camelcase_input	list		None	list or dictionary with lowercase keys

flask.py

Import:

labpack.parsing.flask

Description:

extract_request_details

Signature:
extract_request_details(request_object, session_object=None)

Description:
a method for extracting request details from request and session objects

    NOTE:   method is also a placeholder funnel for future validation
            processes, request logging, request context building and
            counter-measures for the nasty web

Argument	Type	Required	Default	Description
request_object	object	Yes	None	request object generated by flask from request route
session_object	object		None	session object generated by flask from client cookie

extract_session_details

Signature:
extract_session_details(request_headers, session_header, secret_key)

Description:
a method to extract and validate jwt session token from request headers

Argument	Type	Required	Default	Description
request_headers	dict	Yes	None	dictionary with header fields from request
session_header	str	Yes	""	string with name of session token header key
secret_key	str	Yes	""	string with secret key to json web token encryption

validate_request_content

Signature:
validate_request_content(request_content, request_model, request_component=”body”)

Description:
a method to validate the content fields of a flask request

Argument	Type	Required	Default	Description
request_content	dict	Yes	None	dictionary with content fields to validate
request_model	object	Yes	None	object with jsonmodel class properties
request_component	str		"body"	string with name of component of request evaluated

grammar.py

Import:

labpack.parsing.grammar

Description:

join_words

Signature:
join_words(word_list, operator=”and”, quotes=False)

Description:

Argument	Type	Required	Default	Description
word_list	NoneType	Yes	None
operator	str		"and"
quotes	bool		False

section_text

Signature:
section_text(text_string, max_characters=500, continue_text=”…”)

Description:

Argument	Type	Required	Default	Description
text_string	NoneType	Yes	None
max_characters	int		500
continue_text	str		"..."

shell.py

Import:

labpack.parsing.shell

Description:

a package of functions for parsing STDOUT and STDERR

convert_table

Signature:
convert_table(shell_output, delimiter=”\t|\s{“, }’, output=”dict”)

Description:
a method to convert a STDOUT shell table into a python data structure

Argument	Type	Required	Default	Description
shell_output	str	Yes	""	string from STDOUT with headers
delimiter	str		"\t|\s{"	string with regex pattern delimiting headers
}'	NoneType	Yes	None
output	str		"dict"	string with type of structure to output (dict, list or csv)

performlab.py

Import:

labpack.performance.performlab

Description:

repeat

Signature:
repeat(function, kwargs, title, count, verbose=True)

Description:

Argument	Type	Required	Default	Description
function	NoneType	Yes	None
kwargs	NoneType	Yes	None
title	NoneType	Yes	None
count	NoneType	Yes	None
verbose	bool		True

randomlab.py

Import:

labpack.randomization.randomlab

Description:

random_binary

Signature:
random_binary(length)

Description:

random_bytes

Signature:
random_bytes(length)

Description:

random_characters

Signature:
random_characters(character_set, length)

Description:

Argument	Type	Required	Default	Description
character_set	NoneType	Yes	None
length	NoneType	Yes	None

random_double

Signature:
random_double(low, high)

Description:

Argument	Type	Required	Default	Description
low	NoneType	Yes	None
high	NoneType	Yes	None

random_fraction

Signature:
random_fraction()

Description:

random_integer

Signature:
random_integer(low, high)

Description:

Argument	Type	Required	Default	Description
low	NoneType	Yes	None
high	NoneType	Yes	None

random_number

Signature:
random_number(length)

Description:

random_object

Signature:
random_object()

Description:

random_shuffle

Signature:
random_shuffle(item_list)

Description:

ip.py

Import:

labpack.records.ip

Description:

describe_ip

Signature:
describe_ip(ip_address, source=”whatismyip”)

Description:
a method to get the details associated with an ip address

Argument	Type	Required	Default	Description
ip_address	NoneType	Yes	None
source	str		"whatismyip"

get_ip

Signature:
get_ip(source=”aws”)

Description:
a method to get current public ip address of machine

settings.py

Import:

labpack.records.settings

Description:

compile_settings

Signature:
compile_settings(model_path, file_path, ignore_errors=False)

Description:
a method to compile configuration values from different sources

    NOTE:   method searches the environment variables, a local
            configuration path and the default values for a jsonmodel
            object for valid configuration values. if an environmental
            variable or key inside a local config file matches the key
            for a configuration setting declared in the jsonmodel schema,
            its value will be added to the configuration file as long
            as the value is model valid. SEE jsonmodel module.

    NOTE:   the order of assignment:
                first:  environment variable
                second: configuration file
                third:  default value
                fourth: empty value

    NOTE:   method is guaranteed to produce a full set of top-level keys

Argument	Type	Required	Default	Description
model_path	str	Yes	""	string with path to jsonmodel valid model data
file_path	str	Yes	""	string with path to local configuration file
ignore_errors	bool		False	[optional] boolean to ignore any invalid values

ingest_environ

Signature:
ingest_environ(model_path=”“)

Description:
a method to convert environment variables to a python dictionary

load_settings

Signature:
load_settings(file_path, module_name=”“, secret_key=”“)

Description:
a method to load data from json valid files

Argument	Type	Required	Default	Description
file_path	str	Yes	""	string with path to settings file
module_name	str		""	[optional] string with name of module containing file path
secret_key	str		""	[optional] string with key to decrypt drep file

remove_settings

Signature:
remove_settings(file_path, retry_count=10, remove_dir=False)

Description:
a method to remove a file using a child process

    http://www.petercollingridge.co.uk/blog/running-multiple-processes-python
    https://docs.python.org/3.5/library/multiprocessing.html

Argument	Type	Required	Default	Description
file_path	str	Yes	""	string with path to file to remove
retry_count	int		10	integer with number of attempts to remove before error is thrown
remove_dir	bool		False	[optional] boolean to remove empty parent directories

save_settings

Signature:
save_settings(file_path, record_details, overwrite=False, secret_key=”“)

Description:
a method to save dictionary typed data to a local file

Argument	Type	Required	Default	Description
file_path	str	Yes	""	string with path to settings file
record_details	list	Yes	None	list or dictionary with record details
overwrite	bool		False	[optional] boolean to overwrite existing file data
secret_key	str		""	[optional] string with key to decrypt drep file

barrel.py

Import:

labpack.storage.barrel

Description:

main

Signature:
main(args)

Description:

retrieve_values

Signature:
retrieve_values(file_path)

Description:

store_values

Signature:
store_values(file_path, data)

Description:

Argument	Type	Required	Default	Description
file_path	NoneType	Yes	None
data	NoneType	Yes	None