Anubhav’s Cassandra Musing https://docs.microsoft.com/archive/blogs/anubhavk/feed.xml Wed, 17 Apr 2019 03:13:08 GMT en-US hourly 1 Fat clients: Are they useful ? https://docs.microsoft.com/archive/blogs/anubhavk/coordinator-nodes-are-they-useful Thu, 18 May 2017 14:44:47 GMT https://blogs.msdn.microsoft.com/anubhavk/?p=165 Cassandra: This is rather straightforward. You can add a setting "JOIN_RING=FALSE" in cassandra-env.sh on a node that isn't joined the ring, yet. If you run 2.1.13, you also need to disable auth in cassandra.yaml since there is a known issue. Once the node joins the cluster, you can verify that its acting purely as coordinator through "nodetool gossipinfo" as below: /10.183.1.25 generation:1494524175 heartbeat:38808 LOAD:38658:221423.0 SCHEMA:14:744cfd95-9066-3f2f-a5e4-6f7c75426db6 DC:72:BatchAnalytics RACK:12:RACBat2 RELEASE_VERSION:4:2.1.13.1131 INTERNAL_IP:6:10.183.1.25 RPC_ADDRESS:3:10.183.1.25 X_11_PADDING:38717:{"workload":"Cassandra","active":"true","health":0.935385733708455} SEVERITY:38807:0.0 NET_VERSION:1:8 HOST_ID:2:8bb16daa-9e6f-4423-a151-ec80e43033d8 TOKENS: not present The logs also print a friendly message that the node hasn't joined the ring yet. If you run "nodetool status" on regular nodes, the coordinator nodes don't show up. Spark We found out that in Scala + Java world, we must change both Scala and Java drivers for white listing to work properly. This is because coordinator nodes don't exist in system.peers and all drivers remove nodes that don't exist in system.peers, by design. The change is straightforward in both drivers, and we will share our changes soon. Benchmarking We ran a Spark job that does 20K writes / sec to our cluster. In regular mode, we need 40 nodes to handle this level of load for our latency requirements. Since we run real-time analytics workloads, we typically see 100K events processed in 20 seconds end-to-end. We were interested in seeing if we can reduce the number of nodes with this mode, and still meet our latency requirements. With prior experiments, we knew that a 20 node cluster does not meet our write latency requirements and puts high load / CPU on all nodes (it takes 30+ seconds for same number of input events). Therefore, we created a 20 node cluster and ran the same load once with all 20 nodes in regular mode, and another time with 5 of them as coordinators. We found out the following. osload cpu There is significant reduction in OS load on regular nodes (16 to 2, specifically). The coordinator nodes show higher load average (8), yet its better than when run in regular mode. The CPU also reduces by 10%. However, the write latency as perceived by the Spark job (or seen via Cassandra metrics) does not change. Conclusion This mode doesn't help us with latency improvement, so it doesn't help our specific app. We decided to not invest in this mode further. It appears that if your App can live with some degraded latency this mode lets you throw more traffic on same number of nodes, thus helping dollar costs.  ]]> Migrating Cassandra cluster to Azure Premium Storage (at scale!) https://docs.microsoft.com/archive/blogs/anubhavk/migrating-cassandra-cluster-to-azure-premium-storage-at-scale Fri, 03 Mar 2017 10:17:57 GMT https://blogs.msdn.microsoft.com/anubhavk/?p=116 Background Office 365 uses Cassandra to learn deeply about its users. Our cluster has 150 TB data and runs on 300 nodes. We use Azure D14 VMs running Ubuntu, and the data is stored on locally attached SSDs. We use spark jobs running in Azure HDI to compute insights on the "big data" stored in Cassandra cluster.  We query with LOCAL_QUORUM and use a RF = 5. This architecture works great at scale, however it does impose the following restrictions / problems.
  1. Azure loses SSDs when it moves VMs (by design). This means we incur a data loss and the node must be hydrated from other nodes (automatically via the built in auto_bootstrap = true setting or explicitly via "nodetool rebuild"). We have observed that none of these approaches work reliably at scale, and we always risk losing some data. The operation also takes multiple days, and if another node goes down we start losing availability.
  2. We can not scale up / down VMs without losing data. This restricts our ability to test scaling in production.
  3. With RF = 5, we pay extra dollar cost. This becomes a real issue as the data size grows
To solve the problems listed above, we started investigating Azure Premium Storage. This is a latest offering from Azure that provides network attached SSD storage with a local SSD cache. The highest SKU is DS 14, and biggest disk is P30 (1 TB). With appropriate RAID configurations, Azure provides linearly scalable IOPS (1 disk provides 5000, so a RAID-0 array of 3 disks provides 15000). There is plenty of documentation online on how this can help run things at scale with reliability as key tenet. Current Architecture original_cluster We have two Casandra rings as seen in the picture. "Analytics" runs heavy batch jobs, and "Cassandra" runs streaming jobs. Each ring contains 150 nodes and each node holds ~600 GB data. With Premium storage, we could reduce RF = 3. So we set a goal to decommission these rings and build two new rings each with 75 "dense" nodes (~900 GB) running on Premium Storage. Initial Experimentation We wanted to perform detailed tests as well as collect anecdotal evidence that Premium Storage will work for us. For anecdotal evidence, we replaced 4 nodes in the heavy Analytics ring with nodes running Premium Storage (one node at a time) and ran them for 2 weeks. We observed the following to confirm Premium Storage wasn't a deal breaker right away.
  1. Time to rebuild / auto-bootstrap the node
  2. Read / Write Latency
  3. Dropped messages / mutations
  4. CPU / Memory pressure
All of above did not show any significant deviation from SSD based nodes. This gave us confidence in the technology. Prototyping The first decision was how many P30 disk are needed on a node. So we ran "iostat -x 2" on 10 nodes in both rings and collected the numbers over 5 days. For our scenarios, iops average around 5000, and peaked to 10000 infrequently. This meant that we needed 3 P30 disks in a RAID-0 configuration. Since each P30 disk gives 1 TB, this would also satisfy our size requirements per node.   analytics_iops cassandra_iops An important takeaway in this process was that the iostat must be run with really high fidelity to capture true IOPS. Since we have bursty loads, running iostat every 5 seconds did not capture the true IOPS (it shows numbers < 1000). With this information we created a small cluster with appropriate RAID-0 setup, and used  Cassandra-Stress to generate load. We observed the same metrics called out above, and ensured that the cluster behaved well under load. Execution We created a brand new ring named BatchAnalytics with 75 nodes. We tried to hydrate data into this ring using "nodetool rebuild". This is where we hit our first problem. During the rebuild process, the number of SSTables grew in huge numbers (typically they are at 2000, and grew beyond 100000 during rebuild). This caused the node to go down with OutOfMemory exceptions. We tried changing concurrent_compactors, stream_throughput but that did not help. The yaml setting that did the trick was - memtable_cleanup_threshold. Changing this from the default of 0.2 to 0.7 made sure that we did not flush data from memory often and gave node much needed room to run compactions. This allowed us to reliably hydrate nodes in ~3 days. Note this setting is absent in the yaml file by default, so it took us quite some time / online searches to arrive at the right knob.  Datastax support helped, too. We use both Size Tiered and Date Tiered compaction strategies. A side-effect of Date Tiered Strategy during rebuilds is it doesn't compact tables quickly enough. This is probably because the "time-series" nature of data that Date Tier relies heavily on breaks during rebuild (when vnodes are used). We changed some tables to Size Tiered in the process so that the SSTables are crunched quickly, and the nodes are able to serve queries with expected latency. We moved our heavy batch jobs to this ring and observed for 2 weeks. This is where we hit our second problem. The problem was that few nodes in the ring consistently died with OutOfMemoryException and a high SSTable count. As far as we knew, our load was evenly distributed (Write Requests, Read Requests did not now major deviation from other nodes), so we could not tell why this happened only on a few nodes. The nodes under question did not have abnormal wide partitions either. As mitigation, we had to manually delete SSTables and incur data loss to keep the nodes running. We worked with Azure support (which is fantastic) to understand if storage was playing a role. Their tools did show that 5% of our operations were getting throttled, and this percentage increased on the "problematic" nodes before they died. We learnt that the throttling works somewhat differently than how we perceived it to be. Azure throttles at every 50 ms interval, so even if you are well below the 5000 IOPS limits bursty operations will lead to throttling. With this information in mind, we added a new node to the ring with 10 P30 disks and gathered same data. Surprisingly, we saw the same throttling on this node. It seemed as if the more storage you give, the more the node would want to push. We observed CPU, load average and GC pauses on all nodes during this process and observed that the nodes were under CPU pressure in general. We mainly debugged high CPU using jstack. The node with 10 disks was a data point in same direction. We tried various tuning approaches available online such as Kernel scheduler queue size, none of which helped. We were convinced that reducing the number of nodes from 150 to 75 was a bad idea, and we must add nodes to the ring so we can have apples-to-apples for better troubleshooting. This is where we hit our third problem. To make the ring build process faster, we moved the traffic back to original SSD based ring, added nodes to new ring with auto_bootstrap = false. We chose to build the ring from within (nodetool rebuild BatchAnalytics). When the nodes were built (faster as expected), we observed that the data size was lower than the typical size. This happened because the tokens had already moved to "empty" nodes, and so they could only hydrate partial data. We realized this pretty quickly, and rebuilt the nodes from other rings (like how we normally do). We moved jobs again to the new ring. In couple days, it was clear that we were on the right path. CPU came down (from 70% to 40%), load average came down (from 30% to < 10%), and GC pauses generally went away. We observed the ring for 3 weeks to confirm that the problem has indeed gone away. With proof that the heavy ring works reliably for 3 weeks, we created the second ring with 150 nodes for Streaming jobs. This ring did not show any problems, as expected. The cluster is now fully migrated to Azure Premium Storage. We have also enabled disc encryption in the process, and haven't seen any issues with it. Takeaways
  1. Cassandra compute is very CPU sensitive. You need to add nodes to cluster to give it more CPU before having to give it more memory, disk etc.
  2. It is crucial to give changes bake time before moving forward.
  3. At scale, most of the typical debugging arsenal (system logs, jstack, top) won't really help. At times, you do need the "tribal knowledge" so you can look at the right places for problems. (In our case, that tribal knowledge happens to be specific gigantic tables and spark jobs)
  4. In hindsight, we did not have strong data to back our decision to reduce number of nodes (which was an added variable besides Premium Storage). This caused hiccups that took time to recover.
  5. Doing functional tests in a small environment followed by doing things in production in crucial to surface issues quickly and make progress. For example, Azure has a limit on how many VMs can live in an availability set -- we could not have found this in any other environment besides production.
  6. Metrics, metrics, metric ! We constantly observed read/write latency, SSTable count and CPU during the whole process to ensure stability of cluster. Having them available in a reliable UX proved very beneficial.
]]> Running Cassandra + Spark at Office 365 https://docs.microsoft.com/archive/blogs/anubhavk/running-cassandra-spark-at-office-365 Wed, 26 Oct 2016 11:11:41 GMT https://blogs.msdn.microsoft.com/anubhavk/?p=115 Real world Repairs https://docs.microsoft.com/archive/blogs/anubhavk/real-world-repairs Mon, 24 Oct 2016 12:12:30 GMT https://blogs.msdn.microsoft.com/anubhavk/?p=95 Background Maintaining anti-entropy in Cassandra is achieved via the repair operation. This is vital, yet not well-supported in the Cassandra world. Thus, one need to think of innovative options to make this work at scale. Following are the tools out there to run repairs on 2.1 clusters. Netflix Tickler: https://github.com/ckalantzis/cassTickler (Read at CL.ALL via CQL continuously :: Python) Spotify Reaper: https://github.com/spotify/cassandra-reaper (Subrange repair, provides a REST endpoint and calls APIs through JMX :: Java) List subranges: https://github.com/pauloricardomg/cassandra-list-subranges (Tool to get subranges for a given node. :: Java) Subrange Repair: https://github.com/BrianGallew/cassandra_range_repair (Tool to subrange repair :: Python) Mutation Based Repair (An idea that is now dropped): https://issues.apache.org/jira/browse/CASSANDRA-8911 Problem Space Office 365 uses Cassandra to learn deeply about its users. We need a reliable way to repair data when cluster has entropy (for variety of reasons). Therefore, we embarked on a journey to evaluate various approaches. We can divide all the datasets into following buckets. High Consistency  system_auth : This table is queried before every CQL query is executed and must be maintained consistent. Medium Consistency These tables have following characteristics
  1. Long TTLs else running repairs is moot.
  2. Power end user scenarios directly (e.g. tables powering APIs)
  3. Need to support Deletes (without TTLs)
  4. Require something beyond "statistically correct" consistency (this goes back to 2, and means "don't run repairs on gigantic tables holding raw data").
The stretch goal here is ability to read at LOCAL_ONE. Experimentation     The tests were ran on a 300 node cluster, and we deleted ~100 GB of data from a node before running various repair approaches. The goal was to see how reliably and quickly the deleted data comes back, and if there is any performance impact on rest of cluster. The default "nodetool repair" simply did not work (even functionally). Various approaches related to subrange repairs brought back the data but they were not reliable. What worked best was a Spark job that ran the table at Consistency=ALL. With enough resources allocated to the job, the job always ran successfully and repaired the data. We strongly recommend Spark as the means to repair if you have it in your stack.]]> Data Modeling Guidelines https://docs.microsoft.com/archive/blogs/anubhavk/data-modelling-guidelines Wed, 05 Oct 2016 09:06:27 GMT https://blogs.msdn.microsoft.com/anubhavk/?p=75 This gives good in depth coverage if everything that cqlsh syntax provides. At times, you may want to pre-aggregate / compute the data before its fed into Cassandra to reduce the number of duplicate copies to serve different API needs. C* 2.1.13  has a bug around column name optimizations. Make sure your column names are less than 15 characters. Do not use secondary indexes except when the indexed column isn't updated frequently and  does not have a large cardinality. For example, if you have a "status" column that can only take "Success" and "Failure" as valid values and isn't changed often, it is a good candidate for secondary index. Using secondary indexes must be an exception in your data model, and not a norm. Simply create another table with partition keys on the columns that you wanted to do secondary index on to solve such problem. This is a good read on on Spark Cassandra connector works: https://www.slideshare.net/RussellSpitzer/spark-cassandralocality Pick correct compaction strategy. We have not used Leveled Compaction yet. For any data that looks like time series, use Date Tiered Compaction Strategy. For other types of data, use Size Tiered Compaction Strategy. Do not try to reason about how DTCS creates time buckets, that code is incredibly hard to understand.   Do not use Allow Filtering. Instead, change your data model and potentially duplicate. Do not use queries with IN in Partition Key, else you will run into really bad performance: https://lostechies.com/ryansvihla/2014/09/22/cassandra-query-patterns-not-using-the-in-query-for-multiple-partitions/ Think about spreading your data across partitions so that no single server gets hot. You can add things like month of a date field, device that the user is using and such to the partition key for this to happen. Use TTL on your tables. Avoid explicit deletes (CQL delete from Foo) statements if possible. Don't use C* distributed counters. There are known issues with using them. EBay has some good articles on data modelling. Read this and this. The sum of all data within a specific partition should be < 100MB. You can adjust TTL or Partition Key to make this happen. Please avoid adding timestamp to partition key. Unless you have enough other elements to it, this will create hotspots in cluster. Use static columns to model one to many relationships. Always set partition+clustering key values when setting static column value. Note that you can't add static columns to an existing schema, so it can be a prohibitive design choice. Prefer prepared statements over simple statements.]]> Debugging Multi-Node Cassandra Cluster on Windows https://docs.microsoft.com/archive/blogs/anubhavk/debugging-multi-node-cassandra-cluster-on-windows Thu, 25 Aug 2016 14:03:24 GMT https://blogs.msdn.microsoft.com/anubhavk/?p=45 First : Run a single node cluster, and set breakpoints.
  1. Download latest version of cassandra from https://github.com/apache/cassandra. Click "Clone in Desktop" button on the right side of the page for quickest download experience.
    1. You can also setup "Egit" so that you can get source control in Eclipse (Just like Team window in VS). Follow: Eclipse GIT Integration
    2. If you want to edit sources, it's better to clone the repo (From command prompt, type "git clone -b <branch name> https://git-wip-us.apache.org/repos/asf/cassandra.git <branch name>". This will create a <branch name> folder in current folder, and download sources in there.)
  2. Download latest version of Eclipse - https://eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/mars/R/eclipse-java-mars-R-win32-x86_64.zip
  3. Download Java 1.8 - https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  4. Download latest version of Ant (1.9.6)- https://ant.apache.org/bindownload.cgi
  5. Download latest version of Maven (3.3.3) - https://maven.apache.org/
  6. Download Python version 2.7 (this is important: Don't download the latest) - https://www.python.org/download/releases/2.7/
  7. For items downloaded above from 2 through 6, be sure to set their executables in your PATH environment variable. It's also a good idea to create JAVA_HOME environment variable and point it to your JDK installation.
  8. Follow instructions from here - https://wiki.apache.org/cassandra/RunningCassandraInEclipse to build Cassandra. If you run into issues, make sure your JDK installation is good (such as "javac -version" should show you 1.8)
  9. Once Eclipse is fired up, you may see errors in building the stress tool. Remove that from project build settings.
To do so, right click Project -> Select Properties, java Build Path. Ignore what the link in (8) says for Main class, instead choose "org.apache.cassandra.service.CassandraDaemon" as the Main class.
    1. In Debug -> Debug Configurations menu, you need to double click on "Java Application" to select a new configuration.
    2. Provide following VM arguments
For Cassandra 2.1.9 (current version in DSE 4.7.3 cluster) -Dcassandra.config=file:E:\DI\cassandra\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.port=10020 -com.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcassandra.storagedir=F:\data For Cassandra 3.x -Dcassandra.config=file:E:\DI\cassandra\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.ssl=false -Dcassandra.jmx.local.port=7199 -Dcassandra.storagedir=F:\data Replace E:\DI\Cassandra with your location. F:\data can be any directory. For JMX pick a port that's not busy [You can use wintellect Process Explorer for this]
  1. If you have Cassandra running as Datastax services, stop them from services.msc
  2. Click "Debug". At this point, the view will change and you should see bunch of output in Console window at the bottom.
  3. Note the logs are written to build\test\logs directory by default.
Run cqlsh.bat from E:\DI\Cassandra\bin. It should successfully connect and you should be able to run commands. You can also connect from Dev Center if you like. Running a command like below cqlsh> CREATE KEYSPACE Test1 WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' :  3 }; Should print below in eclipse output. INFO  20:05:45 Create new Keyspace: KeyspaceMetadata{name=test1, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[], functions=[], types=[]} INFO  20:05:45 Enqueuing flush of keyspaces: 1492 (0%) on-heap, 0 (0%) off-heap   To stop cassandra, click on "Stop sign" icon on Console window on the right.   You're all set to put breakpoints and be a C* wizard now. Now you are ready to debug a multi-node cluster 1. Create two copies of Cassandra (say E:\cassandra-bkup and E:\cassandra). These will be our two nodes in a cluster. 2. We will use the fact that 127.0.0.1 through 127.0.0.8 map to localhost. The first node will run as 127.0.0.1 and second as 127.0.0.2 In E:\cassandra-bkup Edit cassandra.yaml from E:\cassandra-bkup\conf Replace "localhost" with "127.0.0.1" Replace seeds: "127.0.0.1" with  seeds: "127.0.0.2" Use the following VM args to Eclipse. -Dcassandra.config=file:E:\DI\cassandra-bkup\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra-bkup\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.1 -Dcassandra.jmx.local.port=7199 -Dcassandra.storagedir=E:\DI\cassandra-bkup\data Start Cassandra in Eclipse and make sure its running fine. In E:\cassandra Edit cassandra.yaml from E:\cassandra\conf Replace "localhost" with "127.0.0.2" Use the following VM args to Eclipse. -Dcassandra.config=file:E:\DI\cassandra\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.2 -Dcassandra.jmx.local.port=7200 -Dcassandra.storagedir=E:\DI\cassandra\data Run "nodetool status" from both E:\cassandra-bkup\bin and E:\cassandra\bin and you will see this => Datacenter: datacenter1 ======================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving --  Address    Load       Tokens  Owns    Host ID                               Rack UN  127.0.0.1  139.12 KB  256     ?       d24dfee8-05b8-4ba0-b200-2e432d648b6d  rack1 UN  127.0.0.2  123.92 KB  256     ?       9e7fe5dc-a3e8-4695-88e4-b2edbac4a113  rack1 Now you can put breakpoints and bring nodes up / down and see how it all works. FAQ: What if I get an error that another process is listening on port 10020 (or some other port that Cassandra complains about) ? Follow this and delete that process: https://stackoverflow.com/questions/48198/how-can-you-find-out-which-process-is-listening-on-a-port-on-windows]]> Test your Cassandra Knowledge https://docs.microsoft.com/archive/blogs/anubhavk/the-cassandra-challenge Mon, 22 Aug 2016 21:05:19 GMT https://blogs.msdn.microsoft.com/anubhavk/?p=25 The Quiz 1. have one data center in my setup, and I have setup a replication factor of 5 for my keyspace. If I query data using LOCAL_QUORUM consistency, how many nodes can go down before I'll start seeing data loss ? Will this change if I change replication factor to 6 ? If yes, to what value ? 2. Suppose I have setup replication=3 for my setup, and I read and write using LOCAL_QUORUM. Initially all nodes had a value of "10" for a piece of data. When I wrote a new value of "20", I got an error back suggesting "Could not achieve LOCAL_QUORUM". Now, if I read the value using LOCAL_QUORUM would I get an error, "10", "20", or something else ? What if hinted handoff is enabled ? 3. Suppose I have setup replication=3 for my setup, and I read and write using LOCAL_QUORUM. Let us assume that all 3 nodes holding my data have different values (because repairs did not run / data went missing / whatever other reason). Will I get a LOCAL_QUORUM failure on read ? 4. We specify contact points when establishing a Cassandra connection. If those nodes go down afterwards, does the connection keep working ? What if the nodes are seed nodes ? 5. How can I tell if my keyspace is evenly distributed across the datacenter ? How can I tell what nodes hold a particular piece of data ? 6. I have one keyspace set to replicate in two data centers. I changed it to not replicate in one data center. Now, I want to remove a node in this data center. Should I use "nodetool removenode" or "nodetool decommission" ? Which one would be faster and why ? 7. Is it necessary to run a "nodetool repair" after doing a "nodetool removenode", what about "nodetool decommission" ? 8. Suppose a node in a data center is down. Now, I want to remove another node. Will I be able to remove successfully if I use consistency of LOCAL_QUORUM for my read / write queries ? 9. I am running a "nodetool rebuild" to build my node from scratch. Unfortunately, it gives up in the middle. If I run it again, will the node try to fetch the same data again from other nodes ? 10. I am replacing a node in the cluster using -Dcassandra.replace_address=X in cassandra-env.sh. How can I tell the time it will take to completely join the ring ? 11. I am replacing a node in the cluster using -Dcassandra.replace_address=X in cassandra-env.sh. Until the node joins the ring completely, will it serve any read / write requests ? 12. I am replacing a node in the cluster using -Dcassandra.replace_address=X in cassandra-env.sh. Once the node joins the ring, do I need to run a "nodetool repair" ? If yes, why ? If no, why not ? 13. I am replacing a node in the cluster using -Dcassandra.replace_address=X in cassandra-env.sh. Is it correct this entry can only be removed after the node joins the ring (meaning, if the node reboots in the middle and the entry does not exist, it won't be able to proceed) ? 14. "nodetool" commands run on the node where started and don't have any effect on other nodes. What is the exception to this behavior ? 15. Which of the following commands are safe to run multiple times without unexpected side-effects / performance implications ? A) nodetool repair B) nodetool cleanup C) nodetool snapshot D) nodetool removenode 16. Suppose we add a new node to an existing data-center. Once the node has completely bootstrapped, some data from old node must have moved over to this node to balance the ring. Thus, we should see disks from old nodes getting free. Is this correct ? 17. Does everything in Cassandra run at same priority ? 18. How can I tell how many threads are allocated for handling write requests, read requests and compactions ? What about repairs and streaming ? 19. What is the best way to reduce the number of SS Tables present on a node ? A) nodetool compact B) increase concurrent_compactors in yaml C) nodetool cleanup D) nodetool repair 20. Explain the difference between /var/log/cassandra/output.log and /var/log/cassandra/system.log. 21. Suppose OpsCenter is down, and you need to find out approximately how much data Cassandra node is receiving per minute. What can you do ? 22. When OpsCenter shows that a node is down ("grey"), does this mean DSE is down or DSE is up but CQLSH is down ? 23. What is a "schema mismatch" ? How do you detect it, and get out of it ? 24. Suppose a node goes down and moves to another rack. When it comes back up, we update cassandra-rackdc.props to tell Cassandra about it. However, Cassandra won't start up if you change racks in a running cluster. Why ? 25. Suppose we have a 2 data-center setup with same number of nodes in each DC and we are using random partitioner with vnodes. Is it safe to assume there must exist a node in both DCs that have same tokens (therefore same data if such a replication is set) ? 26. You're seeing high number of SS Tables for a table on a node. What could be possible reasons for this ? A] Low concurrent_compactors in yaml B] The table received a lot of data only on this node C] Poor Compaction properties defined on the table while creation D] All of the above How can you confirm / deny [B] ? Can you think of any other reasons ? 27. Suppose we are running DSE Spark and Cassandra on a node (like how we do on all of our nodes). At any point, how many Java processes would be running on the box ? Why ? 28. You want to tell CQLSH to execute a query on a given Cassandra node. How would you do that ? Will setting CONSISTENCY to ONE on the node from where you started CQLSH do the job ? 29. In what situations the tokens assigned to nodes change ? A] When a new node joins the ring B] When a node leaves the ring C] When "nodetool repair" is run D] When a node goes down for long period of time 30. Suppose a table uses Size Tiered Compaction Strategy. What is a typical file layout you will see on disk ? 31. When running "nodetool cleanup", you notice that the disk usage is going high at times. Why would this happen ? 32. How will you determine if a node has "wide partitions" for a table ? How will you fix such a "wide partition" ? 33. Will "Truncate Table" work if any nodes in the cluster are down ? 34. Why is re-creating a table (drop followed by create) problematic ? 35. Which of the following statements are true ? A] Seed nodes act as coordinators for Cassandra clients B] Ops Center uses seed nodes to talk to the Cluster C] Once setup, seed nodes cannot be changed D] All of the above 36. Is it possible to disable CQLSH Auth on a node temporarily ? If so, how ? 37. If a few nodes are down in the cluster, and you attempt to add a new node you may get an error like below. Can you concretely explain what this is about ? java.lang.RuntimeException: A node required to move the data consistently is down (/xx.xx.xx.xx). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false 38. What ports does Cassandra use ? 39. Will Cassandra move data on dynamic properties such as load on nodes, nodes going up / down (meaning if you run "nodetool repair" will Cassandra move data from nodes that are now down to other nodes to increase availability?) 40. Can you rename a data center on a cluster ? What about renaming a cluster ? 41. As you know, SSTableLoader and CQLSH import are the two ways to load data into Cassandra. Besides the fact that CQLSH does not work for large amounts of data, is there something else that makes it a non-viable choice for a dataset ? 42. Suppose a requirement came up, where you need to export all the data written from now onward somewhere else. There are no timestamp columns on your tables, so you can't query data that way and use CQLSH export. Can you think of any other way to achieve this ? 43. Suppose your data pattern is such that you never update an existing row. In such a situation, does compaction add any value at all (since there are no "duplicates" so to speak) ? 44. You're seeing a large number of Dropped Mutations on a node in Ops Center. Which of the following statement are true about this key metric ? A] The number reported is per node. B] The number will be persisted even if you restart DSE. C] The number represents the number of reads that failed D] All of above 45. Which of the following may fix the high number of Dropped Mutations ? A] Increase write request timeout setting in yaml B] nodetool setstreamthroughput 0 C] nodetool repair D] None of above 46. You deleted some row in a table. That deleted data magically started re-appearing after some time in your read queries. What can be the possible causes for this ? A] Some nodes went down and did not come up until after the gc_grace_seconds setting on the table B] You deleted using consistency LOCAL_QUORUM but read the row later with consistency ONE C] The data did not get deleted in the first place. Cassandra dropped the mutation, but never returned an error back to the client. D] Any of above 47. Secondary indexes are usually frowned upon. Which of the following are the reasons for it ? A] They are "local" to the node. So, Cassandra has to talk to all nodes to figure out the column you're asking for. B] They need to be rebuilt every time the column value changes, which is costly C] Compactions don't run on secondary index tables as often as other tables, thus they may take disk space D] All of above 48. Suppose you’re using vnodes and random partitioner. If you now read data in a table in a paged fashion, will you get the data in same order every time ? Why or why not ? 49. Which of the following statements are true ? A] If Cassandra gets a read and write request at the same time, the write request gets higher priority. B] The only way to generate new SS Tables on disk is through memtable flush. C] Repairs run continuously in the background D] All of above E] None of above 50. If a node goes down, which of the following will try to bring it up ? A] Datastax Ops Center Agent B] Linux Kernel C] Some other nodes through gossip D] None of above 51. How can you tell how much CPU compaction is consuming ? 52. Does "nodetool removenode" result in any data streaming ? The Answers Answer 1 3 nodes need to go down before you will start seeing issues. Here is why: https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_config_consistency_c.html Answer 1.1 Same as 1. Answer 2 It depends. Let's assume the nodes are called N1, N2 and N3. When the write request failed, let's assume it wrote data to N1, but could not write to N2 and N3 (thus failing QUORUM). When the read request comes, if it falls on N1, you will get "20" back and N2,N3 will get fixed as part of Read Repair. However, if it falls on N2 or N3 and N1 is down, Cassandra has no way of knowing that N2 and N3 have stale data, thus you will get "10" back. Answer 2.1 If Cassandra was able to replicate the data to N2 or N3 before you issued a Read, you will get "20" back. Answer 3 No, you will get the latest result. QUORUM fails iff the node does not respond back in allotted time. Cassandra has no way of determining if the data is "correct" given its distributed nature. Answer 4 Yes. The client get updates from the cluster as nodes go up and down. However, if you restart the client and try establishing a new connection using the nodes that are now down, it will not work. Answer 4.1 Seed nodes only play a role for gossip. So, it doesn’t matter if they go down - everything will continue to work as is assuming the cluster is healthy in general. Answer 5 nodetool status <keyspace> and make sure the percentage on every node is equal https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsStatus.html Answer 5.1 nodetool getendpoints Note, this will tell what nodes should hold the data. It will not tell if they actually hold the data. The logic of this command simply runs the math that does the token computation. https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsGetEndPoints.html Answer 6 Either. Since, the node does not own the keyspace there is nothing to stream out to other nodes when you run "nodetool decommission". Normally, decommission streams the data owned by the node to other nodes thus taking a lot of time. Therefore in this case you can do a decommission if DSE process is up else "removenode". Both will take equal amount of time. Answer 7 "nodetool decommission" streams the tokens owned by the node to other nodes as well as the data meaning the cluster continues to stay consistent. "nodetool removenode" does the same thing, but nodes other than the one being removed move data around. Answer 8 The answer has nothing to do with your consistency. It depends on if the node that is already down needs to own some of the token ranges that the leaving node owns. Obviously, Cassandra is smart to give token ranges to other nodes, so you should not see any problems in a reasonably big cluster. Answer 9 Yes, unfortunately. Answer 10 Run "nodetool status" and check the size on other nodes. The node that is being replaced needs to get to that size. Byt running "nodetool status" a few times, you will get a feel for the rate at which the node is pulling data, thus helping you compute the total time. Answer 11 No. Answer 12 No. The node simply replaced an existing node, so the token arrangement hasn't changed. Running a repair is necessary only when the data corresponding to tokens needs to move to appropriate nodes. Answer 13 No. Once the node figures out it needs to replace itself, this is written to one of the system tables. Thus, you can remove the entry as soon as you see "JOINING THE RING" in system.log Answer 14 "nodetool repair". Read more here: https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRepair.html Note any commands that show information about other nodes (e.g. status, ring, gossipinfo etc) are all really just showing cached information on the node where you're running nodetool. Answer 15 "nodetool removenode" is the only command that's safe for re-entry. All other commands will have varying level of impact. "repair" if run multiple times will cause too much data movement (if not actual, at least for merkle tree computation). "cleanup" is fine to run multiple times, but it will also cause unnecessary work on the node Answer 16 No. Unless a "nodetool cleanup" is run, the data does not get deleted even though the node doesn't own it. This is by design (you can feel free to call it questionable design if you like). Answer 17 Everything runs at the NORM_PRIORITY except compactions and repairs (they run at LOW_PRIORITY) Answer 18 Generally you can't tell. This is hardcoded for various stages. However, for compactions you can configure this through a yaml setting called "concurrent_compactors" Answer 19 B is the safest way to do so. A is strongly advised against as it will create one giant SS Table that will then create problems for future compactions. C and D will most likely not lead to any success unless the node is indeed imbalanced. Answer 20 When DSE starts is spews stuff in output.log for first minute or so. Then everything goes to system.log (which includes the stuff in output.log). For practical purposes, output.log is important if the process can't start after doing a "sudo dse service start" as it will tell why it could not start. Answer 21 Look at the size of files in /mnt/cassandra/commitlog. Here is a good refresher on what commitlog is: https://stackoverflow.com/questions/34592948/what-is-the-purpose-of-cassandras-commit-log Answer 22 CQLSH is down. Therefore, it is a great way of telling if the node is functional. Answer 23 This happens when the schema changes don't propagate to all nodes. Every node generates a hash based on the schema it knows, and if all nodes agree this must match. When nodes are down and schema changes are made, they may not make it to all nodes. Similarly, if you re-create tables it will lead to unexpected mismatches. This can be detected using "nodetool describecluster", and fixed using "nodetool resetlocalschema" or restarting the node or a combination of both. Answer 24 Cassandra uses rack information to distribute the load to different racks. Thus changing a rack messes up with this logic, and they have safe guarded by not letting you change it on the fly. You can tell Cassandra to ignore the rack change using -Dcassandra.ignore_rack. However, you must run a "nodetool repair" after this so that Cassandra correctly places replicas. It is a good mental exercise to think why a repair is needed here. Answer 25 No. As much as it is unintuitive, it is not implemented that way. Answer 26 Any / all of the above. You will need to eliminate one thing at a time. Answer 26.1 You can add a graph in Ops Center that shows "Write Requests" and compare the numbers with other nodes. Answer 26.2 No TTL on the table. If using DTCS, extremely low max_sstable_age. Answer 27 1 (Cassandra started as sudo service dse start) + Number of spark executors + Datastax Agent + Datastax Agent Monitor Answer 28 There is no way to achieve this reliably. While you can experiment with CONSISTENCY ONE, Cassandra may still pick a different coordinator every time. Answer 29 A and B. Repair makes sure that the nodes that are expected to hold data for its token range holds it, it never changes the token assignment. Answer 30 Couple of really big files, a few medium sized and 3/4 really small ones. Answer 31 As part of cleanup, Cassandra is reading existing SS Tables and writing new ones with the unnecessary data removed. Thus, you may notice a little bump in disk usage while this completely settles. Answer 32 "nodetool cfstats" will show you Maximum Partition Bytes. If this number is greater than 100 MB, it would be considered a "wide partition". You will need to change the schema of your table and add more entries to the Partition Key to fix this. You can not do anything once the schema is made to fix this. Answer 33 No. It requires CONSISTENCY ALL. Answer 34 This is really a bug in Cassandra. If a node is down / does not get the schema update when DROP TABLE is run, it will continue to use the old ID for the table after the new table is created. When it receives writes / reads for this table, it results into ID mismatch causing "couldn't find cfid" exceptions in system logs and eventually system instability. Answer 35 Correct answer is "None of the above". The only function seed nodes provide is, all nodes provide frequent gossip updates to them. Answer 36 Yes. Adjust auth related settings in yaml. Answer 37 When you add a node it will take over the range of an existing node, and thus it needs to stream data from it to maintain consistency. If the existing node is unavailable, the new node may fetch the data from a different replica, which may not have some of data from the node which you are taking the range for, what may break consistency. For example, imagine a ring with nodes A, B and C, RF=3. The row X=1 maps to node A and is replicated in nodes B and C, so the initial arrangement will be: A(X=1), B(X=1) and C(X=1) Node B is down and you write X=2 to A, which replicates the data only to C since B is down (and hinted handoff is disabled). The write succeeds at QUORUM. The new arrangement becomes: A(X=2), B(X=1), C(X=2) (if any of the nodes is down, any read at QUORUM will fetch the correct value of X=2) Now imagine you add a new node D between A and B. If D streams data from A, the new replica group will become: A, D(X=2), B(X=1), C(X=2) (if any of the nodes is down, any read at QUORUM will fetch the correct value of X=2) But if A is down when you bootstrap D and you have -Dcassandra.consistent.rangemovement=false, D may stream data from B, so the new replica group will be: A, D(X=1), B(X=1), C(X=2) Now, if C becomes down, reads at QUORUM will succeed but return the stale value of X=1, so consistency is broken. See CASSANDRA-2434 for more background. Answer 38 By default, Cassandra uses 7000 for cluster communication (7001 if SSL is enabled), 9160 for Thrift clients, 9042 for native protocol clients, and 7199 for JMX. The internode communication, Thrift, and native protocol ports are configurable in cassandra.yaml. The JMX port is configurable in cassandra-env.sh (through JVM options). All ports are TCP. Answer 39 No Answer 40 You cannot rename a data center. You can however rename the cluster. Answer 41 CQLSH export / import does not support preserving TTLs. Answer 42 You can take a snapshot of data using "nodetool snapshot" and then enable "incremental_backups" in yaml. This will make sure that all the new SS Tables are being written to another folder (under snapshots) that you can then easily copy and load somewhere else. It is a hack, but a good one ! Answer 43 You would still need some process to expire data based on TTL. Since compaction is that process today, you would need compactions. Answer 44 A] is true. Answer 45 A] assuming you don't have any other cluster issue Answer 46 A] is the most likely cause for this. Here is a good reading: https://wiki.apache.org/cassandra/DistributedDeletes B] will also cause this, but the hope is you don't do this normally Answer 47 A] and B] Answer 48 Yes. The data is returned in the Murmur3 hash order on the partition key, and then sorted on clustering key. Answer 49 Correct answer is E] Cassandra does not have a notion of priority, so both write and read requests will be handled in the same priority. New SS Tables can be generated through repairs or some nodetool commands such as cleanup. Repairs don't run by default. Answer 50 D] There is no built-in mechanism to re-start a node that went down. Answer 51 Run "sudo iostat" and look at the %nice column Answer 52 Interestingly yes. All nodes that now are responsible for the tokens of removed node try to fetch necessary data from other nodes in the cluster. This can lead to increase in CPU and SSTables on such nodes. Code :https://github.com/apache/cassandra/blob/d384e781d6f7c028dbe88cfe9dd3e966e72cd046/src/java/org/apache/cassandra/service/StorageService.java#L2548  ]]>