sparkstreaming支持多种流输入，like kafka, flume, twitter, zeromq or plain old tcp sockets，并且可以在上面进行transform操作，最终数据存入hdfs，数据库或dashboard
spark streaming is an extension of the core spark api that allows enables high-throughput, fault-tolerant stream processing of live data streams.
internally, it works as follows. spark streaming receives live input data streams and divides the data into batches, which are then processed by the spark engine to generate the final stream of results in batches.
before we go into the details of how to write your own spark streaming program, let’s take a quick look at what a simple spark streaming program looks like.
streamingcontext是sparkstreaming的入口，就像sparkcontext对于spark一样
其实streamingcontext就是sparkcontext的封装，通过<code>streamingcontext.sparkcontext</code>可以取得
参数除了batchduration，其他的都和sparkcontext没有区别
a <code>streamingcontext</code> object can be created by using
the <code>appname</code> is a name of your program, which will be shown on your cluster’s web ui.
the <code>batchinterval</code> is the size of the batches, as explained earlier.
additionally, the underlying sparkcontext can be accessed as <code>streamingcontext.sparkcontext</code>.
通过图示很清晰的说明什么是dstream，和基于dstream的transform是怎样的？
discretized stream or dstream is the basic abstraction provided by spark streaming. it represents a continuous stream of data, either the input data stream received from source, or the processed data stream generated by transforming the input stream. internally, it is represented by a continuous sequence of rdds, which is spark’s abstraction of an immutable, distributed dataset. each rdd in a dstream contains data from a certain interval, as shown in the following figure.
these underlying rdd transformations are computed by the spark engine. the dstream operations hide most of these details and provides the developer with higher-level api for convenience. these operations are discussed in detail in later sections.
dstream支持transformations and output，和spark的action不太一样
there are two kinds of dstream operations - transformations and output operations.
similar to rdd transformations, dstream transformations operate on one or more dstreams to create new dstreams with transformed data.
after applying a sequence of transformations to the input streams, output operations need to called, which write data out to an external data sink, such as a filesystem or a database.
transformations大部分都和rdd中的一样，参考原文，重点说下底下的几个，
updatestatebykey operation
用流数据持续更新state，比如下面的例子，wordcount
只需要定义updatefunction，如何根据newvalue更新现有的state
the <code>updatestatebykey</code> operation allows you to maintain arbitrary stateful computation, where you want to maintain some state data and continuously update it with new information.
to use this, you will have to do two steps.
define the state - the state can be of arbitrary data type.
define the state update function - specify with a function how to update the state using the previous state and the new values from input stream.
let’s illustrate this with an example. say you want to maintain a running count of each word seen in a text data stream. here, the running count is the state and it is an integer. we define the update function as
transform operation
transform可以在dstream上apply任意的rdd操作，尤其有意思的是，可以通过transform使用spark的mllib和graph算法
the <code>transform</code> operation (along with its variations like <code>transformwith</code>) allows arbitrary rdd-to-rdd functions to be applied on a dstream. it can be used to apply any rdd operation that is not exposed in the dstream api. for example, the functionality of joining every batch in a data stream with another dataset is not directly exposed in the dstream api. however, you can easily use <code>transform</code> to do this. this enables very powerful possibilities. for example, if you want to do real-time data cleaning by joining the input data stream with precomputed spam information (maybe generated with spark as well) and then filtering based on it.
window operations
作为streaming处理，当然需要支持典型的slide-windows based的方法
可以看下面的图，可以将original中的slide-windows上的多个rdds生成windowed dstream上的单个rdd
两个重要的参数，windows length和slide inteval，下面的图中分别是3和2
finally, spark streaming also provides windowed computations, which allow you to apply transformations over a sliding window of data. this following figure illustrates this sliding window.
as shown in the figure, every time the window slides over a source dstream, the source rdds that fall within the window are combined and operated upon to produce the rdds of the windowed dstream. in this specific case, the operation is applied over last 3 time units of data, and slides by 2 time units. this shows that any window-based operation needs to specify two parameters.
window length - the duration of the window (3 in the figure)
slide interval - the interval at which the window-based operation is performed (2 in the figure).
these two parameters must be multiples of the batch interval of the source dstream (1 in the figure).
给个例子，统计过去30秒的word count，每次递进10秒
some of the common window-based operations are as follows. all of these operations take the said two parameters -windowlength and slideinterval.
output operations
when an output operator is called, it triggers the computation of a stream. currently the following output operators are defined:
dstream的persistence是通过persist之上的每个rdd来实现的，并没有什么特别的地方
需要注意的是，window-based operations会自动将persist设为true，因为slide-windows based算法中，每个rdd都会被读多遍
而且对于来自kafka, flume, sockets, etc.的数据，default的persist策略是，产生2个replicas并同时写入两个节点，便于fault-tolerance
spark的fault-tolerance取决于replay，对于普通的spark数据是从文件读的，所以replay只是重新读取数据而已，很简单
但对于流数据，数据一旦流过就无法replay，所以sparkstreaming必须在source采取2-replicas的策略
similar to rdds, dstreams also allow developers to persist the stream’s data in memory. that is, using <code>persist()</code> method on a dstream would automatically persist every rdd of that dstream in memory. this is useful if the data in the dstream will be computed multiple times (e.g., multiple operations on the same data). for window-based operations like <code>reducebywindow</code> and<code>reducebykeyandwindow</code> and state-based operations like <code>updatestatebykey</code>, this is implicitly true. hence, dstreams generated by window-based operations are automatically persisted in memory, without the developer calling <code>persist()</code>.
for input streams that receive data over the network (such as, kafka, flume, sockets, etc.), the default persistence level is set to replicate the data to two nodes for fault-tolerance.
同样dstream也支持checkpoint，即把rdd写入hdfs中，当然cp会影响相应时间，所以要处理好这个balance，一般cp interval为5 - 10 times of sliding interval，至少10秒
在dstream中cp主要用于stateful operation，比如前面说的wordcount，由于streaming处理的特点，即使persist，也只可能保留一段时间的数据，所以如果不去做cp，crash后是无法replay出太久以前的结果的，所以对于dstream而已cp的意义更为重要（对于spark，cp只是节省时间）
a stateful operation is one which operates over multiple batches of data. this includes all window-based operations and the<code>updatestatebykey</code> operation. since stateful operations have a dependency on previous batches of data, they continuously accumulate metadata over time. to clear this metadata, streaming supports periodic checkpointing by saving intermediate data to hdfs. note that checkpointing also incurs the cost of saving to hdfs which may cause the corresponding batch to take longer to process. hence, the interval of checkpointing needs to be set carefully. at small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. typically, a checkpoint interval of 5 - 10 times of sliding interval of a dstream is good setting to try.
to enable checkpointing, the developer has to provide the hdfs path to which rdd will be saved. this is done by using
the interval of checkpointing of a dstream can be set by using
for dstreams that must be checkpointed (that is, dstreams created by <code>updatestatebykey</code> and <code>reducebykeyandwindow</code> with inverse function), the checkpoint interval of the dstream is by default set to a multiple of the dstream’s sliding interval such that its at least 10 seconds.
在性能优化上，主要考虑两点
1. 如果降低每个batch的执行时间，增加并发度，减少不必要的serialization和降低task launch的overhead（减少tasksize和使用standalone or coarse-grained mesos mode ）
2. 设置合适batch size，batch size越小肯定越接近流处理，但是overhead越大，所以需要balance，建议是先在一个比较保守的size和比较小的流量下进行测试，然后如果系统稳定，在逐步减少size和加大流量
getting the best performance of a spark streaming application on a cluster requires a bit of tuning. this section explains a number of the parameters and configurations that can tuned to improve the performance of you application. at a high level, you need to consider two things:
reducing the processing time of each batch of data by efficiently using cluster resources.
setting the right batch size such that the data processing can keep up with the data ingestion.
reducing the processing time of each batch
level of parallelism
data serialization
the overhead of data serialization can be significant, especially when sub-second batch sizes are to be achieved. there are two aspects to it.
serialization of input data: to ingest external data into spark, data received as bytes (say, from the network) needs to deserialized from bytes and re-serialized into spark’s serialization format. hence, the deserialization overhead of input data may be a bottleneck.
task launching overheads
if the number of tasks launched per second is high (say, 50 or more per second), then the overhead of sending out tasks to the slaves maybe significant and will make it hard to achieve sub-second latencies. the overhead can be reduced by the following changes:
task serialization: using kryo serialization for serializing tasks can reduced the task sizes, and therefore reduce the time taken to send them to the slaves.
these changes may reduce batch processing time by 100s of milliseconds, thus allowing sub-second batch size to be viable.
setting the right batch size
for a spark streaming application running on a cluster to be stable, the processing of the data streams must keep up with the rate of ingestion of the data streams. depending on the type of computation, the batch size used may have significant impact on the rate of ingestion that can be sustained by the spark streaming application on a fixed cluster resources. for example, let us consider the earlier wordcountnetwork example. for a particular data rate, the system may be able to keep up with reporting word counts every 2 seconds (i.e., batch size of 2 seconds), but not every 500 milliseconds.
24/7 operation
streaming操作，数据是无限的，所以必然需要cleanup metadata，比如generated rdd
可以在sparkcontext创建前，通过<code>spark.cleaner.ttl</code> 设置cleanup时间
注意cleanup时间必须要比任意window operation的窗口大小要长
this value is closely tied with any window operation that is being used. any window operation would require the input data to be persisted in memory for at least the duration of the window. hence it is necessary to set the delay to at least the value of the largest window operation used in the spark streaming application. if this delay is set too low, the application will throw an exception saying so.
monitoring
memory tuning
concurrent garbage collector: using the concurrent mark-and-sweep gc further minimizes the variability of gc pauses. even though concurrent gc is known to reduce the overall processing throughput of the system, its use is still recommended to achieve more consistent batch processing times.
简而言之，spark的fault-tolerance是很容易做的，因为只有source data存在，任何rdd都是可以被replay出来的
所以关键问题在于如何保证source data的fault-tolerance
in this section, we are going to discuss the behavior of spark streaming application in the event of a node failure. to understand this, let us remember the basic fault-tolerance properties of spark’s rdds.
an rdd is an immutable, deterministically re-computable, distributed dataset. each rdd remembers the lineage of deterministic operations that were used on a fault-tolerant input dataset to create it.
if any partition of an rdd is lost due to a worker node failure, then that partition can be re-computed from the original fault-tolerant dataset using the lineage of operations.
since all data transformations in spark streaming are based on rdd operations, as long as the input dataset is present, all intermediate data can recomputed. keeping these properties in mind, we are going to discuss the failure semantics in more detail.
failure of a worker node
对于worker node的失败，只需要换个worker把rdd给replay出来就可以了，前提就是要能够重新取到source data
所以对于用hdfs作为input source的case，fault-tolerance是没有问题的
关键是，以network作为input source的case（比如，kafka，flume，socket），如何保证能够重新读到source data？
做法是首先以2-replicas把source data放到两个node的memory中，一个fail，可以用另一个replay
但是一种情况下，还是会丢数据，就是当network receiver所在的node fail的时候，那么从network上读到并还没有来得及完成replicas的数据就会丢失
另外需要注意的是，对于rdd的transform是可以保证exactly-once semantics的，但是对于output operations就只能保证at-least oncesemantics
比如，某个rdd已经部分被写到file中，这个时候fail，当worker replay出rdd的时候会重新写一遍
所以如果要保证exactly-once，必须自己加上additional transactions-like mechanisms
there are two failure behaviors based on which input sources are used.
using hdfs files as input source - since the data is reliably stored on hdfs, all data can re-computed and therefore no data will be lost due to any failure.
using any input source that receives data through a network - for network-based data sources like kafka and flume, the received input data is replicated in memory between nodes of the cluster (default replication factor is 2). so if a worker node fails, then the system can recompute the lost from the the left over copy of the input data. however, if the worker node where a network receiver was running fails, then a tiny bit of data may be lost, that is, the data received by the system but not yet replicated to other node(s). the receiver will be started on a different node and it will continue to receive data.
since all data is modeled as rdds with their lineage of deterministic operations, any recomputation always leads to the same result. as a result, all dstream transformations are guaranteed to have exactly-once semantics. that is, the final transformed result will be same even if there were was a worker node failure. however, output operations (like <code>foreachrdd</code>) have at-least once semantics, that is, the transformed data may get written to an external entity more than once in the event of a worker failure. while this is acceptable for saving to hdfs using the <code>saveas*files</code> operations (as the file will simply get over-written by the same data), additional transactions-like mechanisms may be necessary to achieve exactly-once semantics for output operations.
failure of the driver node
对于driver node failure ，spark是没有做任何处理的，job会失败
而对于sparkstreaming而言，这样是无法接受的，因为流数据你无法重新读出的，对于流处理必须要保证24/7
所以这里做的更多，会定期把dstreams的metadata chechpoint到hdfs上，这样当driver node failure，被restart后，可以冲checkpoint文件中把状态恢复过来，继续执行
但是当前对于以network作为input source的情况下，数据还是会丢失，由于driver node的fail会导致sourcedata丢失，虽然你metadata是checkpoint了
to allow a spark streaming program to be recoverable, it must be written in a way such that it has the following behavior:
when the program is being started for the first time, it will create a new streamingcontext, set up all the streams and then call start().
when the program is being restarted after failure, it will re-create a streamingcontext from the checkpoint data in the checkpoint directory.
note: if spark streaming and/or the spark streaming program is recompiled, you must create a new <code>streamingcontext</code> or<code>javastreamingcontext</code>, not recreate from checkpoint data. this is because trying to load a context from checkpoint data may fail if the data was generated before recompilation of the classes. so, if you are using <code>getorcreate</code>, then make sure that the checkpoint directory is explicitly deleted every time recompiled code needs to be launched.
for other deployment environments like mesos and yarn, you have to restart the driver through other mechanisms.
recovery semantics
there are two different failure behaviors based on which input sources are used.
using any input source that receives data through a network - the received input data is replicated in memory to multiple nodes. since, all the data in the spark worker’s memory is lost when the spark driver fails, the past input data will not be accessible and driver recovers. hence, if stateful and window-based operations are used (like<code>updatestatebykey</code>, <code>window</code>, <code>countbyvalueandwindow</code>, etc.), then the intermediate state will not be recovered completely.
in future releases, we will support full recoverability for all input sources. note that for non-stateful transformations like <code>map</code>,<code>count</code>, and <code>reducebykey</code>, with all input streams, the system, upon restarting, will continue to receive and process new data
本文章摘自博客园，原文发布日期： 2014-02-21