Data lake FAQ

This topic describes some commonly asked questions (FAQ) about data lake and provides solutions to these issues. Some metrics mentioned in this topic can be obtained only from the profiles of the SQL queries. To obtain the profiles of SQL queries, you must specify set enable_profile=true.

Slow HDFS DataNodes

Issue description

When you access the data files stored in your HDFS cluster, you may find a huge difference between the values of the __MAX_OF_FSIOTime and __MIN_OF_FSIOTime metrics from the profiles of the SQL queries you run. This indicates that some DataNodes in the HDFS cluster are slow. The following example is a typical profile that indicates a slow HDFS DataNode issue:

 - InputStream: 0
   - AppIOBytesRead: 22.72 GB
     - __MAX_OF_AppIOBytesRead: 187.99 MB
     - __MIN_OF_AppIOBytesRead: 64.00 KB
   - AppIOCounter: 964.862K (964862)
     - __MAX_OF_AppIOCounter: 7.795K (7795)
     - __MIN_OF_AppIOCounter: 1
   - AppIOTime: 1s372ms
     - __MAX_OF_AppIOTime: 4s358ms
     - __MIN_OF_AppIOTime: 1.539ms
   - FSBytesRead: 15.40 GB
     - __MAX_OF_FSBytesRead: 127.41 MB
     - __MIN_OF_FSBytesRead: 64.00 KB
   - FSIOCounter: 1.637K (1637)
     - __MAX_OF_FSIOCounter: 12
     - __MIN_OF_FSIOCounter: 1
   - FSIOTime: 9s357ms
     - __MAX_OF_FSIOTime: 60s335ms
     - __MIN_OF_FSIOTime: 1.536ms

Solution

You can use one of the following solutions to resolve this issue:

• [Recommended] Enable the data cache feature, which eliminates the impact of slow HDFS DataNodes on queries by automatically caching the data from external storage systems to the BEs or CNs of your StarRocks cluster.
• [Recommended] Shorten the timeout duration between the HDFS client and DataNode. This solution is suitable when Data Cache cannot help resolve the slow HDFS DataNode issue.
• Enable the Hedged Read feature. With this feature enabled, if a read from a block is slow, StarRocks starts up a new read, which runs in parallel to the original read, to read against a different block replica. Whenever one of the two reads returns, the other read is cancelled. The Hedged Read feature can help accelerate reads, but it also significantly increases heap memory consumption on Java virtual machines (JVMs). Therefore, if your physical machines provide a small memory capacity, we recommend that you do not enable the Hedged Read feature.

[Recommended] Data Cache

See Data Cache.

[Recommended] Shorten timeout duration between HDFS client and DataNode

Configure the dfs.client.socket-timeout property in the hdfs-site.xml file to shorten the timeout duration between the HDFS client and DataNode. (The default timeout duration is 60s, which is a bit long.) As such, when StarRocks encounters a slow DataNode, the connection request from it can time out within a very short period of time and then be forwarded to another DataNode. The following example sets a 5-second timeout duration:

<configuration>
  <property>
      <name>dfs.client.socket-timeout</name>
      <value>5000</value>
   </property>
</configuration>

Hedged Read

Use the following parameters (supported from v3.0 onwards) in the BE or CN configuration file be.conf to enable and configure the Hedged Read feature in your HDFS cluster.

Parameter	Default value	Description
hdfs_client_enable_hedged_read	false	Specifies whether to enable the hedged read feature.
hdfs_client_hedged_read_threadpool_size	128	Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. This parameter is equivalent to the dfs.client.hedged.read.threadpool.size parameter in the hdfs-site.xml file of your HDFS cluster.
hdfs_client_hedged_read_threshold_millis	2500	Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to 30. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a hedged read against a different block replica. This parameter is equivalent to the dfs.client.hedged.read.threshold.millis parameter in the hdfs-site.xml file of your HDFS cluster.

If the value of any of the following metrics in your query profiles exceeds 0, the Hedged Read feature is enabled.

Metric	Description
TotalHedgedReadOps	The number of hedged reads that are started up.
TotalHedgedReadOpsInCurThread	The number of times that StarRocks has to start up a hedged read in the current thread instead of in a new thread because the Hedged Read thread pool has reached its maximum size specified by the hdfs_client_hedged_read_threadpool_size parameter.
TotalHedgedReadOpsWin	The number of times that a hedged read beats its original read.