- Cassandra is a distributed nosql database without a master node (but there are seed nodes)
- Seed nodes are only used to bootstrapping new nodes, i.e., they provide the source of gossip so the cluster does not have partition of information (where nodes form their own mini clusters and information becomes inconsistent)
- Each node sends gossip message every second to 3 other nodes to tell them what it knows about the cluster.
- New nodes need to contact seed nodes to learn about the cluster. It is crucial that all nodes have the same set of seed nodes configured in cassandra.yaml.
- Dead nodes are marked down and other nodes will store "hinted handoff" for 3 hours by default.
- Replication factor and consistency levels (W + R > replication factor => data consistency)
- Cassandra uses three main protocols: gossip, thrift (called rpc) and binary (called native)
- gossip is for inter-node communication only
- thrift is used by CQLSH and clients to send commands to Cassandra (like initiating connections and authenticating users)
- binary is used for transporting data
Configuring Cassandra Connection
To check the status of a node, use the following commands:
nodetool status // checks the status of the cluster
nodetool info // checks the status of the node
nodetool netstats // checks the network stats of the node
TODO: add some examples
- rpc_host
This is the host name or IP address that CQLSH or client uses to connect to the node. Note in Cassandra 2.0.9 and before, the parameter is also the value in system.peers.rpc_address, which is used by client when doing auto node detection. Because this value must be resolved to a network interface recognised by Cassandra (so it can bind and listen to it), the value can't be a public IP address configured on a gateway. This has caused great trouble when we try to make a client connect via the public IP in AWS VPC.
How Datastax Java client find the nodes:
The Cluster.builder().addContactPoints() is used to add a list of Cassandra nodes that will be contacted when the client starts to figure out the cluster info. Each node is queried to get records from system.peers table, and for each record (together with the node itself, because it's not present in the peers able), the rpc_address is added to the list of Host(s), and Host(s) not in the records will be removed. Note the comparison is done with the record's rpc_address and the Host.address, which will not match if the Host.address is a public IP, and hence the Host will be removed.
Example: Suppose we have three Cassandra nodes
And we call
Node
|
Private
IP
|
Public
IP
|
Node1
|
10.220.0.20
|
54.220.0.20
|
Node2
|
10.220.0.21
|
54.220.0.21
|
Node3
|
10.220.0.22
|
54.220.0.22
|
Cluster.builder().addContactPoints("52.220.0.20", "52.220.0.21", "52.220.0.22");
Then when the client starts, you may see those Hosts, and only 54.220.0.22 is reachable from your client (because it cannot resolve private IP in a different VPC)
54.220.0.22, 10.220.20, 10.220.0.21The only solution is to implement a AddressTranslater class that translates the private IP to public IP.
- broadcast_rpc_host (only available for 2.1 and after)
This is the rpc host name used by Cassandra nodes in auto node detection, described here. It is used by clients to make connections to Cassandra, and can be configured to a public IP, hence solving the problem above.
- broadcast_address
The IP address used by nodes from another DC to make connections to the node. This is to solve the private/public IP issue above, but only works for inter-node communication. Hence you should always set this value to the public IP.
- listen_address
The IP address used by this node to listen for communication from another node. This is like the counterpart of broadcast_address, and should always be set to the private IP address. Like rpc_host, this address has to be recognised by Cassandra.
