Configuration¶
To configure Crail use the *.template files as a basis and modify it to match your environment.
Set the $CRAIL_HOME
environment variable to your Crail deployment’s path.
cd $CRAIL_HOME/conf
mv crail-site.conf.template crail-site.conf
mv crail-env.sh.template crail-env.sh
mv core-site.xml.template core-site.xml
mv slaves.template slaves
Note: Docker containers can be configured by using config files above. However it is only recommended for complex configurations. See Docker for details.
The purpuse of each of these files are:
- crail-site.conf: Configuration of the file system, data tiers and RPC
- crail-env.sh: Allows to pass additional JVM arguments
- core-site.xml: Configuration of the HDFS adapter
- slaves: Used by the start-crail.sh script to ease running Crail on multiple machines
crail-site.conf¶
There are a general file system properties and specific properties for the different storage tiers. Typical properties you might want to change are:
Property | Default Value | Description |
---|---|---|
crail.namenode.address |
crail://localhost:9060 | Namenode hostname and port |
crail.cachelimit |
1073741824 | Size (byte) of client buffer cache |
crail.cachepath |
/dev/hugepages/cache | Hugepage path to client buffer cache |
Advanced properties (Only modify if you know what you are doing):
Property | Default Value | Description |
---|---|---|
crail.directorydepth |
16 | Maximum depth of directory tree |
crail.tokenexpiration |
10 | Seconds write token is valid |
crail.blocksize |
1048576 | Size (byte) of block |
crail.user |
crail | Username used for HDFS adapter |
crail.debug |
false | Enable debug output |
crail.statistics |
true | Collect statistics |
crail.rpctimeout |
1000 | RPC timeout in milliseconds |
crail.datatimeout |
1000 | Data operation timeout in milliseconds |
crail.buffersize |
1048576 | Size (byte) of buffer (buffered stream) |
crail.slicesize |
524288 | Size (byte) of slice (transfer unit) |
crail.singleton |
true | Only create a single instance of the FS |
crail.regionsize |
1073741824 | Size (byte) of allocation unit (Cache) |
crail.directoryrecord |
512 | Size (byte) of directory entry |
crail.directoryrandomize |
true | Randomize iteration of directories |
crail.cacheimpl |
org.apache.crail.memory.MappedBufferCache | Client buffer cache implementation |
crail.namenode.fileblocks |
16 | File |
crail.namenode.blockselection |
roundrobin | Block selection algorithm: roundrobin or random |
RPC¶
Crail’s modular architecture allows to plugin different kinds of RPC implementations. The crail.namenode.rpctype
property
is used to configure the RPC implementation. We currently offer two implementations:
- A TCP implementation based on narpc (default): org.apache.crail.namenode.rpc.tcp.tcpnamenode
- A RDMA implementation based on darpc: org.apache.crail.namenode.rpc.darpc.DaRPCNameNode
Logging¶
To allow shutting down the namenode without loosing data Crail offers namenode logging.
It can be enabled by setting a path to the log file with crail.namenode.log
.
Note: this feature is experimental and should be used with caution
Storage Tiers¶
Crail offers multiple types of datanode dependent on your network and storage requirements:
- TCP storage tier backed by DRAM (default)
- RDMA storage tier backed by DRAM
- NVMe over Fabrics storage tier, typically backed by NVMe drives
Crail allows to use multiple storage tier types together, e.g. to store hot data on
DRAM and cold data on NVMe, or extend your DRAM by NVMe storage. Storage types can be
configured as a comma separated list by setting the crail.storage.types
property:
- TCP: org.apache.crail.storage.tcp.TcpStorageTier
- RDMA: org.apache.crail.storage.rdma.RdmaStorageTier
- NVMf: org.apache.crail.storage.nvmf.NvmfStorageTier
Each of the storage types in the list defines a storage class, starting from storage class 0.
Types can appear multiple times to allow defining multiple storage classes for a type.
The maximum number of storage classes needs to be specified with the
crail.storage.classes
property (default = 1).
In the default configuration storage classes are used in incremental order, i.e.
storage class 0 is used until no more space is left then storage class 1 is used and so on.
However filesystem nodes (e.g. files) can also be created on a particular storage class and
can be configured to inherit the storage class of its container. The default storage
class of / is 0 however it can be configured via crail.storage.rootclass
.
Storage tiers send keep alive messages to the namenode to indicate that they are still
running and no error has occured. The interval in which keep alive message are
send can be configured in seconds with crail.storage.keepalive
.
Some of the configuration properties can be set via the command line when starting a storage tier. Refer to Run for details.
TCP Tier¶
The TCP storage tier (org.apache.crail.storage.tcp.TcpStorageTier) is backed by DRAM. The following properties can be set to configure the storage tier:
Property | Default Value | Description |
---|---|---|
crail.storage.tcp.interface |
eth0 | Network interface to bind to |
crail.storage.tcp.storagelimit |
1073741824 | Size (Bytes) of DRAM to provide, multiple of allocation size |
crail.storage.tcp.datapath |
/dev/hugepages/data | Hugepage path to data |
Advanced properties:
Property | Default Value | Description |
---|---|---|
crail.storage.tcp.port |
50020 | Port to listen on |
crail.storage.tcp.allocationsize |
crail.regionsize | Allocation unit |
crail.storage.tcp.queuedepth |
16 | Data operation queue depth (single connection) |
crail.storage.tcp.cores |
1 | Threads to process requests |
RDMA Tier¶
The RDMA storage tier (org.apache.crail.storage.rdma.RdmaStorageTier) is backed by DRAM. The following properties can be set to configure the storage tier:
Property | Default Value | Description |
---|---|---|
crail.storage.rdma.interface |
eth0 | Network interface to bind to |
crail.storage.rdma.storagelimit |
1073741824 | Size (Bytes) of DRAM to provide; multiple of allocation size |
crail.storage.rdma.datapath |
/dev/hugepages/data | Hugepage path to data |
Advanced properties:
Property | Default Value | Description |
---|---|---|
crail.storage.rdma.port |
50020 | Port to listen on |
crail.storage.rdma.allocationsize |
crail.regionsize | Allocation unit |
crail.storage.rdma.localmap |
true | Use mmap if client is colocated with data tier |
crail.storage.rdma.queuesize |
32 | Data operation queue depth (single connection) |
crail.storage.rdma.type |
passive | Operation type: passive or active (see DiSNI) |
crail.storage.rdma.persistent |
false | Allow restarting a data tier if namenode logging is used |
crail.storage.rdma.backlog |
100 | Listen backlog |
crail.storage.rdma.connecttimeout |
1000 | Connect timeout in milliseconds |
NVMf Tier¶
The NVMf storage tier (org.apache.crail.storage.nvmf.NvmfStorageTier) is typically backed by NVMe drives. However some target implementations support using any block device. Unlike the RDMA and TCP storage tier the NVMf storage tier is not involved in any data operation but only is used to provide metadata information. Crail uses the jNVMf library to connect to a standard NVMf target to gain metadata information about the storage and provide the information to the namenode. Clients directly connect to the NVMf target. Crail has been tested to run with the Linux kernel, SPDK and Mellanox ConnectX-5 offloading target.
The following properties can be set to configure the storage tier:
Property | Default Value | Description |
---|---|---|
crail.storage.nvmf.ip |
localhost | IP/hostname of NVMf target |
crail.storage.nvmf.port |
50025 | Port of NVMf target |
crail.storage.nvmf.nqn |
nqn.2017-06.io.crail:cnode | NVMe qualified name of NVMf controller |
crail.storage.nvmf.namespace |
1 | Namespace of NVMe device |
Advanced properties:
Property | Default Value | Description |
---|---|---|
crail.storage.nvmf.allocationsize |
crail.regionsize | Allocation unit |
crail.storage.nvmf.queueSize |
64 | NVMf submission queue size |
crail.storage.nvmf.stagingcachesize |
262144 | Staging cache size (byte) for read-modify-write operations |
crail-env.sh¶
Modify crail-env.sh to pass additional JVM arguments to crail
respectively
start-crail.sh
.
It is recommended to increase heap (e.g. -Xmx24g
) and young generation heap size
(e.g. -Xmn16g
) for the namenodes and TCP datanodes to improve performance
for large deployments.
core-site.xml¶
To configure the HDFS adapter modify core-site.xml. For example the Crail shell
crail fs
uses the HDFS adapter thus requiring the core-site.xml file to
be setup. Modify fs.defaultFS
to match crail.namenode.address
in
crail-site.conf. The default is:
<property>
<name>fs.defaultFS</name>
<value>crail://localhost:9060</value>
</property>
slaves¶
The slaves file can be used to ease starting Crail on larger deployments. Refer to Run for details. Each line should contain a hostname where a storage tier is supposed to be started. Make sure the hostname allows passwordless ssh connections. Note that the hostnames are not used by the storage tier itself but only by the start/stop-crail.sh scripts to start and stop storage tiers. IP/hostname of the storage tiers or any other configuration option are either passed by command line arguments or via crail-site.conf. Command line arguments can be configured in the slaves file following the hostname.