Category Archives: BigData

[MQ,CEP] 0. 개요 및 구조 정의

MQ라는 것에 맨 처음 관심을 갖게된 계기는..

지금의 분류로는 빅데이터로 볼 수 있는 증권데이터였다

 

목표는 실시간 분석이었는데 그 때도 SQL로 처리하는 것은 포기했었다.

어차피 시간순서대로 사용되는 데이터니 파일로 저장하고 순차적으로 쓰려고 했었는데 실시간 분석이 안되는게 또 문제라서 중간 버퍼에 메서드를 필터로 깔아놓고 돌리는 방법을 썼었다.

당시 뭐 설계 구조도 좀 안좋았고 해서 속도도 안나오고 해서 포기했었는데(지금 보니 코드도 안좋다)

요즘 빅데이터나 실시간 분석 솔루션이 많이 나오면서 다시 관심을 갖게 됐다.

 

당시 원했던 구조는

  1. 세개가 연결된 구조 : 진입 ==> 필터(스크립트언어, java osgi) ==> 탈출
    ==>>==>>==>>==>>==>>==>>
    in[] ==> brane[] ==>out[]
  2. control api : 1번을 조작하는 api

추가로 생각한 부분은

in -> grabber, receiver

out -> shotter, deliverer

filter->aggregator, wiper

 


요즘 나와있는 툴

  1. fluentd는 보니 번은은 구현 돼 있다

http://www.fluentd.org/

 

2. kafka

http://kafka.apache.org/

 

3. logstash

https://www.elastic.co/products/logstash

 

4. flume

https://flume.apache.org/

 

5. rabbitmq

https://www.rabbitmq.com/

 

 

LogBase 순차데이터 저장소

필요이유:
데이터의 형태에 따라 퇴적의 데이터 저장소가 있다. 요즘처럼 RDBMS의 용량을 초과할 정도의 대량의 데이터가 나오는 경우에는 이 필요성이 더 커진다.
순차데이터저장소는 증권,날씨,시스템 로그 등처럼 순차적으로만 조회되고
그냥 로그를 쌓는것과 형태는 같다. 단지 이것을 관리해주는 기능만 추가될 뿐이다.
나와있는 오픈소스들의 기능과 연동해서 사용한다면 만들어야 할 기능은 많지 않다. 아예 없을지도…?
마이크로소프트 아치브데이터와 같은 것을 사용하지 않는 것은 하둡 클러스터에 얹는것을 기본 목표로 하기 때문.
hbase가 충분한 성능만 보여준다면 여기에 그냥 입력하는 입력기 정도만 만드는 수준으로 끝내도 될지도 모르겠다.

적합한 데이터의 형태 :
각 데이터별로 확실한 스키마가 있고
대량의 데이터가 실시간으로 쏟아져나오며
조회시에도 중간중간의 몇몇 값을 조회하는게 아니라 전체적으로 순차적으로 분석을 해야하는
증권데이터가 가장 적합하다.

파일의 저장형태:
yyyymmdd(hh)정도까지면 충분하지않을까? 확장가능하도록 처리하는게 좋을듯하다. 스키마/시간 – 계층형 디렉토리 구분
스키마의 루트에 dtd파일로 스키마 정의, 시작시에 스키마를 메모리에 로딩
key-value, 구분자
배치처리를 이용한 중복데이터 제거.
MQ를 이용한 데이터 수신

Hadoop, Pig 기본설치 실행법

현재 서버 상태
CentOS 6.3
OpenJdk 1.6
ant

Hadoop 설치

http://hadoop.apache.org
에서 다운로드
http://www.apache.org/dyn/closer.cgi/hadoop/common/

$ cd /opt

1. 다운로드 처리
$ wget http://apache.tt.co.kr/hadoop/common/hadoop-1.1.2/hadoop-1.1.2-bin.tar.gz

2. 압축해제 후 디렉토리명 변경
$ tar zxvf hadoop-1.1.2-bin.tar.gz
“hadoop-1.1.2” 디렉토리 생성됨
$ mv hadoop-1.1.2 hadoop

3. 자바패스 추가
$ vi /etc/profile
—– —– —– —–
맨 자바 패스 설정 추가 bash_profile를 사용해도 좋은데 모르면 그냥 따라하기
export JAVA_HOME=/usr/lib/jvm/java-openjdk
export PATH=$PATH:$JAVA_HOME/bin
—– —– —– —–
$ source /etc/profile

4. 하둡설정 편집
$ vi {hadoop_home}/conf/hadoop-env.sh
—– —– —– —–
export JAVA_HOME=/usr/lib/jvm/java-openjdk
—– —– —– —–

5. WordCount 테스트 작업
이 명령어가 수행되도록 작업
$ bin/hadoop jar hadoop-examples-1.1.2.jar wordcount input out1

$ bin/hadoop {jar파일} {처리할 jar파일} {wordcount모듈} {input text file directory} {outputdirectory}
명령어 실행해보면 한단계씩 설명이 잘 나온다
$ bin/hadoop
$ bin/hadoop jar
$ bin/hadoop jar hadoop-examples-1.1.2.jar
이런식으로 한단계씩 입력해보자
위 작업이 테스트까지 잘 수행이 되면 output 디렉토리에 처리된 파일이 올라온다.

Pig 설치
http://pig.apache.org
http://www.apache.org/dyn/closer.cgi/pig

1. 다운로드 처리
$ wget http://apache.mirror.cdnetworks.com/pig/pig-0.11.0/pig-0.11.0.tar.gz

2. 압축해제 후 디렉토리 변경
$ tar zxvf pig-0.11.0.tar.gz
$ mv pig-0.11.0 pig

3. 돼지 패스 추가
$ vi /etc/profile
—– —– —– —–
export PIG_HOME=/opt/pig
export PATH=$PATH:$PIG_HOME/bin
—– —– —– —–

4. 돼지 설정 변경
conf/pig.properties에서는 변경할게 없다
$ vi bin/pig
패스 추가
—– —– —– —–
export HADOOP_CONF_DIR=/opt/hadoop/conf
export PIG_HOME=/opt/pig
export PIG_CONF_DIR=$PIG_HOME/conf
—– —– —– —–

5. 실행 및 테스트
$ bin/pig
제대로 설정했으면 grunt> 쉘로 진입한다

CEP 상용 및 오픈소스 제품 목록

벤더 | 제품명 | 설명
TIBCO | TIBCO BusinessEvents | 업계최고 | http://www.tibco.com/
StreamBase Systems | StreamBase CEP | 사용편리 | http://www.streambase.com/
SAP | Sybase Event Stream Processor | Sybase제품 인수 | http://www.sybase.co.kr/
Oracle | Oracle CEP | Esper기반 | http://docs.oracle.com/cd/E16764_01/doc.1111/e14476/overview.htm
IBM | InfoSphereStreams | 테러방지 목적으로 시작, 멀티미디어 스트림 프로세싱 가능 | http://www-01.ibm.com/software/kr/data/infosphere/streams/
IBM | WebSphere Business Events | – | http://en.wikipedia.org/wiki/IBM_Websphere_Business_Events
Microsoft | StreamInsight | – | ~
Informatica | RulePoint | – | ~

오픈소스 제품
에스퍼 Esper, Marketcetera, Drools, …

라이센스는 드룰즈가 상용으로 사용하기 더 좋아보이는데… 기업체들은 에스퍼를 더 선호하는 것 같다.

Writing Your First EventFlow Application

StreamBase DocumentationGetting Started

Writing Your First EventFlow Application

첫 이벤트플로우 어플리케이션

Purpose 목표

In this topic, you will develop your new MyFirstApp.sbapp EventFlow application.

여기서는 MyFirstApp.sbapp EventFlow application을 만들어보자.

Let’s briefly preview the steps in our application design process. First you will provide streams for events to enter and exit the application. You will define a schema that specifies what field names and data types your input stream expects. Then, you will add one or more operators to process the data. Finally, you will connect these components to determine how data flows through the application.

 

Preliminary Step

Before building your application:

  1. In the StreamBase Studio menu, click Window → Preferences. Expand the StreamBase Studio menu and open the Authoring page. Verify that the default setting, Automatically typecheck components, is enabled.To learn about typechecking, see About Typechecking in this topic.
  2. If your application is not already open in an EventFlow Editor, double-click MyFirstApp.sbapp in the Package Explorer to open it. You can have multiple editors open in this area of your perspective, each with its own tab labeled with the file name.Notice the Palette view under the Package Explorer. Whenever an EventFlow is open and its canvas is displayed, the palette is loaded with icons that represent the components you will use to design your EventFlow application. The icons are organized in drawers of different component types, as shown in the figure below.The Palette view is empty when no EventFlow canvas is displayed.

Building Your First EventFlow Application

Follow this procedure to develop the MyFirstApp.sbapp application.

  1. If your application is not already open in an EventFlow Editor, double-click its name (MyFirstApp.sbapp) in the Package Explorer to open it. You can have multiple editors open in this area of your perspective, each with its own tab labeled with the file name.
  2. Add the input stream and define its schema:
    1. In the Palette view, open the Streams category, then drag an Input Stream icon from the palette into your EventFlow canvas:

      Notice the yellow background of the InputStream1 icon: this color is a signal to you that the component does not typecheck yet — in this case, because there is no schema associated with it.

    2. Click the input stream icon in the canvas. When you click any component in an EventFlow Editor, StreamBase Studio opens its Properties view. In the following example, the Properties view is in the upper left:
    3. In the General tab, change the name of the input stream to TradesIn.
    4. Notice that the Edit Schema tab contains has a red  typecheck error icon. This icon should indicate to you that the tab needs attention, so open the Edit Schema tab. It contains an error message (notice the same red error icon) that attempts to help you fix the problem. In this case, it advises you to add fields to the schema.Click the  Add button twice to add a row for each field in the schema. Enter these values in the rows:
      Field Name Type Description
      symbol string Stock symbol
      quantity int Number of shares

      (The descriptions are optional.)

    5. Apply your changes: click a blank area on the canvas, or click the  Apply and Typecheck button near the top-right area of the Properties View. This forces Studio to typecheck your application now, instead of waiting for it to happen automatically. Note that typechecking does not save the application. You can write the application file to disk at any time with File → Save or Ctrl+S or by clicking the Save button in the main toolbar.When you apply your changes, notice that the background color of your TradesIn icon changes from yellow to white. That means the typecheck problem has been cleared, and that Studio is now aware of the schema for the TradesIn component.
  3. Add and configure a Filter operator:
    1. Open the Operators section of the Palette tab. Drag and drop a Filter operator from the palette onto the canvas. Its default name is Filter1. The canvas should now look similar to this:

      Notice that the Filter operator’s background color is yellow. This is because its typecheck status is unknown: it has not been connected yet to a component with a known and valid schema.

    2. If the Outline view is not open, click Window → Show View and choose Outline. This view provides a tree of all the components in your application. It can be helpful for navigating applications with many components. When your application was empty, its outline contained no instances, only a list of component types. Now, if you expand the Operators and Streams categories you should see your current components:
    3. Double-click the Filter1 instance in the Outline view: this selects the component in your canvas and opens its Properties view.
    4. In the General tab, change the name of the operator to TradeFilter.
    5. Open the Predicate Settings tab. Notice that a row in the Predicates table already exists for output port 1, with predicate expression true. Edit this expression in the Predicate column, and specify:quantity >= 10000predicate is an expression that resolves to either true or false for the values in a tuple. Here, the predicate is true for tuples containing trades of 10000 or more; all other tuples are false. The tuple is passed through the Filter operator only if it evaluates to true; all others are ignored.
    6. Now, let’s change the operation slightly: Instead of throwing away tuples with smaller trades, we want to make it possible to check them. To do this, we can make the Filter operators dequeue those tuples to a separate output stream.To make this change, select the option, Create output port for non-matching tuples.

    At this point, your Filter operator’s Properties view should look like this:

    At run time, the Filter operator will evaluate each tuple as it arrives from the input stream. If you had defined multiple predicates, they would be evaluated in order. The first predicate that returns truecauses the filter operation to occur immediately, and any later predicates are ignored.

    We have only one predicate, so each tuple is dequeued either on output port 1 if true, or on output port 2 (the port created for non-matching tuples) if false.

  4. Connect the input stream to the filter operator.The canvas shows each output port as a small black square on the right side of the component. If there are multiple output ports, the upper-most square represents Port 1; the next square down represents port 2, and so on. Your Filter operator now has an extra output port, which you configured in the preceding step.

    Find the output port on the TradesIn icon. Drag your cursor from that square to the input port (the square on the left side) of the filter operator. StreamBase Studio draws an arc connecting the two components. Arcs represent internal streams within EventFlow applications. When you connect the two components, the input stream and the filter operator backgrounds turn white, indicating that they typechecked successfully. For example:

  5. Add the output streams:
    1. Drag and drop an Output Stream icon from the palette into the canvas. Select the output stream on the canvas to open its Properties view.
    2. In the General tab, change the stream’s name to BigTrades.
    3. In the canvas, draw an arc from the upper output port of the Filter operator to the input port of the output stream. Your EventFlow application should look similar to this:
    4. Add a second output stream. Select the output stream, and in its Properties view, change its name to AllTheRest.
    5. Connect the lower output port of the Filter operator to the AllTheRest output stream.

      Tip

      If your canvas looks a little disorganized, try one or both of these adjustments:

      • To automatically arrange the components neatly on the canvas, click Diagram → Layout Application or press Ctrl+L.
      • To rearrange the arcs around components (for example, if an arc overlaps a component’s name), click Diagram → Reroute Connections.

      The final layout of your EventFlow application should look similar to the following:

  6. Use File → Save, Ctrl+S, or click the Save button to save your StreamBase Application.

About Typechecking

Typechecking is a kind of validation that StreamBase performs on each component of an application. A StreamBase application will not run if it has typecheck errors. One of the key advantages of designing inStreamBase Studio is that typechecking can be performed automatically as you work, so you can avoid accumulating problems.

Conceptually, typechecking occurs from the starting points of the application (usually the left side) — and proceeds to the downstream components (typically those connected on the right). When the typecheck process finds an error condition, the background color of the component turns to red, and downstream components cannot be typechecked yet. Typechecks can fail for a number of reasons, such as:

  • Incorrect operator parameters (for example, if the user edited the EventFlow file’s XML source code directly and entered an invalid parameter name or value).
  • Incorrect use of the expression language, such as an invalid character in a field name, again entered by a user who edited the sbapp XML directly.
  • An operator that does not have a schema declared yet.
  • An operator has a schema declared, but it is in conflict with the prior (upstream) connected operator.

Summary and Next Steps

In this topic you learned how to define a StreamBase schema, add components to your application, connect them together, and define functionality by editing the properties of one component. You also learned something about StreamBase typechecking. Your application doesn’t do much, but it’s finished and ready to run.

Do not close StreamBase Studio. To continue the tutorial, please click Next to proceed to one of the following topics:

  • If you would like to create a StreamSQL application, complete the next topic, Creating Your First StreamSQL Application.It will do exactly the same work as the MyFirstApp.sbapp application, and in subsequent topics, you will be able to run either version of the application with identical results.
  • If you are not interested in using StreamSQL, skip the next topic: click Next one more time to go on to Running Your First StreamBase Application .

 

Creating Your First StreamBase Project

StreamBase DocumentationGetting Started

http://docs.streambase.com/latest/index.jsp

Creating Your First StreamBase Project

Purpose

In this topic, we will create a new StreamBase project. StreamBase Studio uses projects to store applications and their associated files.

Note

Throughout this tutorial, we suggest names for projects, applications, and other resources files that you will create. We will refer to those resources using the suggested names, but you are free to choose other names.

Create a Project and Generate an Application

  1. Make sure that you are in the SB Authoring perspective, which presents the views most commonly used for editing StreamBase applications. The SB Authoring button should be selected in the Perspective bar (typically in the top right corner of the Studio window):SB Authoring를 선택해서 사용하도록.  이게 스트림베이스 쓰는데 가장 기본적인 형태다. SB Authroing 선택버튼은 오른쪽 위에 있음

    If it is not, click the  Open Perspective button and switch to the SB Authoring perspective.

    SB Authoring버튼이 안보이면… 저 그림을 클릭해서 SB Authoring 화면을 선택해라.

  2. Click the  New StreamBase Project button near the top of the SB Authoring perspective.새 스트림베이스 프로젝트 버튼을 클릭…
  3. In the New StreamBase Project wizard Project name field, type MyFirstApp.새로운 프로젝트의 이름을 정하기… 그냥 MyFirstApp으로…
  4. By default, StreamBase Studio creates your project in your current workspace. Accept this setting, as well as the default options to create an empty EventFlow application file.위 화면의 기본셋팅으로…
  5. Under the option to create an EventFlow Application file, check the option, and open it now.옵션칸에 있는 Create empty eventflow application file을 선택하고…
  6. Uncheck the Create a server configuration file option, because we will not be changing the default configuration settings for StreamBase Server.Create default server configuration 을 끄고
  7. Uncheck the StreamBase Client API option, because we will not be adding custom Java components to our first application.Java build path api 뭐 어쩌고 저것도 끄고
  8. Click Finish. Click OK to dismiss the confirmation window.피니쉬~ 그리고 뜨는 메세지는 okThe two stub application files are created. Both have the same name as the project, with .sbapp and .ssql extensions. Both are added to your Package Explorer, and one of them, MyFirstApp.sbapp, is opened in the EventFlow Editor.

    이렇게 하고나면….

    두 개의 어플리케이션 파일이 생성됐다.(이상하네… 난 ssql파일은 생성이 안된다)

     

Launching StreamBase Studio

StreamBase DocumentationGetting Started

http://docs.streambase.com/latest/index.jsp

Launching StreamBase Studio

Purpose 목표

If you have read the first topic, you should know enough of the basic StreamBase terms and concepts to start the tutorial.

앞단원을 읽었으면 스트림베이스에 대해 개념은 잡았겠다.

In the tutorial, we will build and run a very simple application that takes a stream of stock trade data, and selects from them only trades above a certain number of shares. We will also create a feed simulator that enters test data into the running application, so that we can verify whether the application works as expected.

이 튜토리얼에서 주식 거래 데이터를 가지고 아주 간단한 어플리케이션을 만든다.  그리고 주식 몇 개 거래 해 본다.

그리고 feed 시뮬레이터도 만들어본다.

Let’s begin by starting StreamBase Studio.

자 시작~

Prerequisites 필요조건

If you are running the StreamBase Trial kit, a license file was automatically e-mailed to you when you downloaded the kit. The license for the full version is e-mailed to you by a StreamBase Systems representative.

스트림베이스 트라이얼 킷을 쓰고 있다면 라이센스 파일은 자동적으로 이메일로 전송됏을거다.

You can run StreamBase Studio and design applications without a license file, but before you can run or debug applications in Studio, your license file must be copied to the etc directory of your installation. For details, see Configuring a StreamBase License.

라이센스 없이 실행은 되는데… 디버깅은 안된다. 디버깅 전에 라이센스 파일을 설치경로의 etc폴더에 옮겨주라.

Start StreamBase Studio 시작하자

  1. Open the Workspace Launcher: 프로그램 시작~
    On Windows
    On a Windows machine, launch StreamBase Studio in one of the following ways: 

    • On the Start menu, select Start → (All) Programs → StreamBase n.m → StreamBase Studio.
    • From the desktop, double-click the StreamBase Studio desktop icon.
    On Linux
    On a Linux machine, launch StreamBase Studio by entering the sbstudio command at a shell prompt. 

  2. Choose a workspaceA workspace is a directory in your file system that contains the projects, files, and other resources that you create with StreamBase Studio. You might maintain several workspaces, each with a different development focus, or each used with a different release of StreamBase.워크스페이스 정하기. 맘에드는 경로 설정… 기본경로를 그냥 써도 된다.

    Click OK to open StreamBase Studio using the default workspace path. If you are opening StreamBase for the first time, a new workspace is created and initialized with default StreamBase Studio settings. If you are using an existing workspace, all your files and any customized settings are preserved from the last time you used it.

    To specify a non-default workspace, before clicking OK you can click Browse to switch to a different, existing workspace, or enter the name of a new directory to create a new workspace.

Summary and Next Steps

In this topic you have learned how to start StreamBase Studio and choose a workspace. You are ready to create a StreamBase project and application.

Click Next to go to the next topic.

 

StreamBase Terms and Components

StreamBase DocumentationGetting Started

http://docs.streambase.com/latest/index.jsp

————————————————————————————————————————-

Quick Facts: StreamBase Terms and Components

Purpose 목표

When you start using a new product, you often have some basic questions, such as:

처음 제품을 사용할 때 이런 의문을 갖는다 :

  • What is it? 이게 뭔데?
  • How will it help me do my job? 이거 쓰면 뭐가 좋아?
  • What is the development cycle and environment? 개발주기나 사용 환경이 어때?
  • What are the product’s components, and what new terms do I need to understand? 구성품은 뭐뭐가 있고 내가 이해해야 할게 뭐가 있지?

This topic addresses those initial questions in enough depth to understand the various lessons taught in the tutorial. 다양한 튜토리얼을 배우면… 이런 첫 질문에 대한 답을 알 수 있을듯…

————————————————————————————————————————-

What is StreamBase? ??스트림베이스??

StreamBase is a new type of computing platform, designed specifically to meet the performance requirements of high-volume, real-time streaming applications. At its core, StreamBase implements a unique Stream Processing Engine™ named StreamBase Server. In contrast to the traditional database model where data is first stored and indexed and then subsequently processed by queries, StreamBase processes the inbound data while it is in flight, as it streams through the server.

스트림베이스는 새로운 종류의 계산computing 플랫폼으로, 특별히 실시간, 대용량, 어플리케이션에 적합하게 디자인 됐다. 핵심은 Stream Processing Engine™ named StreamBase Server. 기존의 데이터베이스 모델은 먼저 저장을 하고 인덱싱을 한 다음에서야 쿼리를 처리했는데 스트림베이스는 데이터가 날라오는 동안에 처리해서 서버에 넣어준다.

Data arrives on a stream as discrete messages called events. Each message contains a tuple, and is processed by StreamBase Server in real time, using the business logic that you defined. Results are and delivered as they are produced, typically in milliseconds. Events are inherently transient: their data can be stored, but it is optional. In many cases, client programs that you write will consume the processed results in real time, and take appropriate actions based on your code’s direction.

데이터는 event라고 불리는 각각의 message로 전달된다. 각각의 message는 tuple을 포함하고 있고, 이것은 StreamBase Server에서 미리 정의 해 놓은 login대로 실시간 처리된다. 이 데이터는 밀리초 내에 처리된다. Event는 순식간에 저장된다 : 데이터는 저장될 수 있지만… 다 저장되는건 아니다. 데이터 저장하는데 시간이 걸려서 처리속도가 늦어질 수 있기 때문에 이 부분은 개발자가 알아서 하도록 한다.

StreamBase can also connect to an external data source, enabling applications to incorporate selected data into the application flow, or to update the external database with processed information. With StreamBase Enterprise Edition an application can connect to a JDBC-compliant database.

StreamBase는 외부데이터 접근도 가능하다. 정해진 데이터를  application flow에 포함시키거나 처리결과를 외부 데이터베이스에 저장할 수 있다. Enterprise Edition은 JDBC-compliant database에 접속할 수 있다.

The following diagrams illustrate the conceptual differences between the older Outbound Processing model and the new StreamBase Inbound Processing model.

밑의 다이어그램은 기존의 데이터 처리 모델과 StreamBase 데이터 처리 개념을 비교해준다.

Outbound Processing Model Used by Traditional Database Products

Inbound Processing Model Used by StreamBase

In StreamBase applications, events deliver data records called tuples that flow through the query and application processing steps, which may in turn transform this data while it is moving. StreamBase provides a set of operators, data processing units that you can add and configure to apply your business logic on the streaming data. With the operators and a supporting cast of data constructs, you can perform such tasks as:

스트림베이스 어플리케이션에서 event는 tuple이라고 불리는 data record를 가져온다. tuple은 이동과 동시에 쿼리와 어플리케이션 처리과정으로 들어간다.  StreamBase는 set of opreator을 제공한다. data processing unit에서는 streaming data에 네 login을 넣을 수 있다. opreator과 supporting cast of data constructs를 이용해서 이런 작업이 가능하다:

  • Applying aggregating functions to windows of real-time data 윈도우의 실시간 데이터를 통합처리
  • Computing new field values by applying mathematical expressions, adding new fields, or dropping fields, from the data streams
  • Filtering data into separate streams for conditional processing
  • Performing approximate buffered sorts of the real-time data
  • Joining previously split streams of data based on key values
  • Within the StreamBase application’s process, populating a shared data table so that other portions of the application can lookup data based on key values

StreamBase also features a graphical rapid-development environment, called StreamBase Studio. It lets you design, test, and deploy streaming applications. StreamBase users report that by using StreamBase Studio, their development teams are able to build streaming applications in less time and at lower costs, when compared with their efforts to create or redesign streaming applications.

스트림베이스는 StreamBase Studio라는 그래픽 기반으로 빠른 개발환경이 특징입니다. StreamBase Studio는 디자인하고 테스트하고 스트림 어플리케이션을 배치하는데 도움을 줍니다.  사용자들이 말하길…. 옛날에 쓰던걸 스트림베이스로 다시 만들어보니 더 짧은시간에 더 낮은 비용으로 만들수가 있었다고 하네요.

————————————————————————————————————————-

How Will StreamBase Help Me Do My Job? 스트림베이스가 내 일에 어떤 도움을 줄까?

For developers and administrators, StreamBase provides graphical, text-based, and command-line tools to build streaming applications quickly and efficiently. 스트림베이스는 빠르고 효율적.

In StreamBase Studio, a collection of views called the SB Authoring perspective lets you design the StreamBase application either graphically, using the text-based StreamSQL language, or with a combination of both. As part of this design step, you define properties for the operators and other components that will apply your business logic on the inbound data. We’ll discuss the SB Authoring perspective’s features in the next section. 스트림베이스 내의 SB Authoring perspective라는 콜렉션은 그래픽언어와 텍스트기반StreamSQL언어를 사용하여 또는 둘다를 사용하여 개발할 수 있게 해 준다.  설계단계에서 비지니스로직의 인바운드데이타 처리에 사용할 연산자와 컴포넌트들을 선언해야한다. 그럼 우리 다음 시간에 SB Authoring perspective의 특징에 대해 말해볼까?

Whether you are developing the application on Windows or Linux, you can use StreamBase Studio to start a StreamBase Server instance. The StreamBase Server loads the application, processing the inbound data according to your definitions in the operators, and responding to any other requests, such as requests for data from Java, C++, or .NET clients that you write. The StreamBase Client Libraries are documented, and we provide examples of writing enqueuer clients (sometimes called producers) that feed data into the application, and dequeuer clients (sometimes called consumers) that accept data from the application for further processing.

 

Once the server is started, StreamBase Studio provides a second perspective, Test/Debug, that contains tools to enqueue or dequeue data on the running application, or run a debugger to pause the application and step through its processing units, or view performance statistics.

Finally, StreamBase provides Client libraries for Java, C++, and on Windows, .NET, that you can use to develop client applications that enqueue to (or dequeue data from) StreamBase applications.

The following diagram illustrates the StreamBase development cycle and platforms.

StreamBase Development Cycle

————————————————————————————————————————-

StreamBase Components and Terms 스트림베이스 컴포넌트와 규정

So far you have learned that StreamBase Studio has a graphical development environment for building StreamBase applications. Let’s take a closer look at the product’s components and terms.

To begin with, what is a stream? A stream is a sequence of data records called tuples. A tuple is similar to a row in a database table. In a high-volume streaming application, the streams flow into the application and are processed in real time, before any (optional) on-disk storage occurs. The tuples in a stream have a schema, which defines each field’s name, position, data type, and size. A field is simply a named value in a tuple, such as a stock’s Symbol in a trade record. A field is similar to a column in a database table. The following diagram illustrates a stream of data containing tuples:

During the course of processing streaming data, you can add, remove, or modify the fields that comprise the tuples, in a highly dynamic fashion. In other words, the initial schema of the stream does not determine its contents during the entire processing execution. 스트리밍 데이터가 처리되는 동안에 튜플을 구성하는 필드를 신나게~ 더하고 빼고 고치는게 가능하다. 다시말해서…. 스트림의 초기 스키마를 보고 프로세스가 끝난 다음의 형태를 알 수는 없다.

Of course, an important step in building any application is determining which fields will comprise the inbound data, how that data should be organized, named, and typed (examples: string, int, long, boolean, double, timestamp), and usually which field(s) will contain key values. Often, though, you have already made those metadata decisions for your existing applications, and you want to know how to define the schemas for StreamBase. We will introduce the schema definition steps later in this tutorial. For now, notice how in the screen below the schema for an application component is defined:

당연히… 어플리케이션을 만드는 가장 중요한 단계는 어떤필드를 인바운드 데이터에 포함시킬까 하는것이다. 이것들을 어떻게 만들어줄지 string, int, long, boolean 등등 뭘로 저장할지… 키값은 어떻게 될지….

이런것은 너의 어플리케이션에서 이미 정의되어 있겠지?(아닌데요). 다음튜토리얼에서 이런것들을 어떻게 하는지 배워볼까? 먼저 지금은.. 스키마의 어플리케이션 컴포넌트가 어떻게 정의되는가나 봐 둬.. 밑에그림

The sample screen above is a peek at just one of many features in StreamBase Studio, our graphical development environment. Let’s take a step back, though, and learn how data gets into a StreamBase application.  위의 샘플화면은 스트림베이스 그래픽개발환경의 몇 가지 특징을 보여준다.

————————————————————————————————————————-

How Does Data Get into StreamBase? 데이터를 어떻게 스트림베이스에 넣어줄까?

The entry points for a StreamBase application are called input streams. Your application may define one or more input streams. The previous sample screen showed the schema definition for a StockIn input stream. But how is data sent to a named input stream? The options are:

스트림베이스에 데이터를 넣는 구멍은 들어가는 구멍(input streams)이라고 부른다.  너의 어플리케이션은 한개이상의 input streams를 갖겠지…

위의 그림은 보여준다. (뭐를?) 스키마를… 정의된… ~을 위해서 StockIn input stream. 근데 데이터는 어떻게 넣는거지? 옵션은:

  • From an adapter program that performs the conversion from an external source (such as a stock market data feed) into the StreamBase protocol. For example, StreamBase provides an adapter that converts TIBCO® Rendezvous™ messages into StreamBase tuples, and vice versa. 스트림베이스는 TIBCO® Rendezvous™라는 어뎁터를 제공한다. 이걸 통해서 넣고 빼고 한다.
  • From an enqueue client program (Java, C++, or .NET) that submits inbound data to one or more input streams defined in your StreamBase application. A client extends the documented StreamBase Client library to interact with a running StreamBase application. JAVA, C++, or .NET으로 만든 enqueue용 손님client프로그램을 이용해서 만든다
  • From an external database connected to your StreamBase application, using SQL queries to read the data. Currently you can connect to a JDBC database. This method enables your application to work with long-lived (historical) data along with the streaming data from its input streams. 외부 데이터베이스 연결은 JDBC를 통해 SQL쿼리를 이용한다. 역사적 데이터를 이용할 때 쓰는거…
  • And you can use StreamBase commands in terminal windows to perform an even greater level of enqueuing operations with running applications. 커맨드라인을 쓰라는데… 몬소린가 이거…이렇게 하면 성능이 이분 더빨라진다고? – 별 중요한것은 아닌듯… 어차피 이렇게는 안할테니까..
  • While designing your application, you can use a graphical feature of StreamBase Studio, called the Feed Simulation Editor, to submit randomly generated data or pre-recorded data from an input file such as a comma-separated value (CSV) file. The Feed Simulation Editor includes defaults or customized settings to control the rate of inbound test data. Feed Simulation Editor을 통해서… CSV파일같은걸로 가져다 쓴다. – 간편하지만.. 실시간에는 적합하지 않을듯…

————————————————————————————————————————-

How Do I Get Data from StreamBase? 스트림베이스에서 데이터 꺼내쓰려면?

You can define output streams that serve as named exit points in your StreamBase application. There are several ways to perform dequeue operations:

스트림베이스 어플리케이션에서 exit points 를 쓰면된다. 몇 가지 방법이 있는데…:

  • Using an adapter program that can return the processed data from StreamBase to the original external source.
  • Using a dequeue client program that listens for outbound data on the application’s streams. A client can be written in Java, C++, or .NET, extending the documented StreamBase Client library to interact with the running application.
  • Using insert or update queries to an external JDBC-compliant data source that is connected to your StreamBase application.
  • You can also use a graphical feature of StreamBase Studio to view the Application Output.
  • And you can use a StreamBase command to dequeue tuples from streams.

위에 나온 방법과 동일한듯… 반대로 하면 되겠지

The following high-level diagram illustrates the different ways that data can enter and exit a running StreamBase application.

Data In and Out of a StreamBase Application

In the preceding diagram, note that the StreamBase adapters are available separately from the StreamBase base kit, and each adapter has its own installation procedure.

————————————————————————————————————————-

StreamBase Studio Development Environment 개발환경

StreamBase Studio supports both graphical and text-based application development:

그래픽환경과 텍스트 환경을 지원한다.

  • EventFlow applications are XML files that are edited graphically using StreamBase Studio’s EventFlow Editor. 이벤트 플로우 어플리케이션은 XML형태로 돼 있는데 그래픽환경으로 처리
  • StreamSQL applications are coded using a query language to describe the application and query streams of data. StreamSQL Applications are stored in text files with an .ssql extension. 스트림SQL어플리케이션은 query language로 돼 있는데 텍스트환경편집 필요

You can also build parts of applications – modules – that can be included as components in other applications. In fact, any EventFlow or StreamSQL application can be used as a module in another application. 모듈형태로 만들 수 있으니까 쪼개서 만들어서 재활용해라~

————————————————————————————————————————-

Building EventFlow Applications 이벤트 흐름 어플리케이션 만들기

To create an EventFlow application in StreamBase Studio, you can drag and drop the icons for various components from a Palette View to the EventFlow Editor’s drawing canvas. As noted earlier, the components include: 이벤트 플로우 어플리케이션을 만들려면 컴포넌트를 드래그앤드롭하면 돼~ 내가 말 했지? 컴포넌트 이런거가 있어~ …..라는데요.

  • Input streams and output streams. 입구. 출구.
  • Operators that can apply your business logic on streaming data, such as aggregating tuples, or merging tuples, or retrieving data from a table. Each operator performs the work that you specify in its Properties view.
  • Data constructs that store information used by an associated StreamBase operator. For example, Query operators are associated with Query Table data constructs. 데이터를 구성하려면 StreamBase operator을 이용해서 해야돼. 예를들어 Query operators는 Query Table data operator을 모아놓은거야.. .. (이게 몬소린지…)  오퍼레이터를 활용해서 데이터를 축적하라는 말인듯…

We will use some of these components during the tutorial, and you can read more about them in the Authoring Guide.

튜토리얼 하면서 이런 컴포넌트를 다 써 보게 될거야… 설명서에서 컴포넌트에 대해 더 읽어볼 수 있어.

Here’s an illustration that shows the drag and drop step, from the left-side Palette to the EventFlow Editor’s drawing canvas.

드래그앤드롭 하는 방법… ..얼마나how 친절kind 눡덜youare

Once a component is placed on the canvas, you define its runtime behavior by setting parameters and (for most components) defining expressions in the Properties View. In the next screen, we have selected the Update_Bids_and_Asks Query operator in the EventFlow Editor to open its Properties view. In the General tab, we can change the name of the selected operator and add a description that documents our design. We can use the other tabs to define the behavior of the component.

 

————————————————————————————————————————-

Building StreamSQL Applications

StreamSQL어플리케이션 만들기

If you prefer to work with SQL queries instead of boxes and arrows, you can build applications using StreamSQL. StreamSQL will look familiar to SQL programmers, but there are some important differences. While SQL operates on tabular data that is stored somewhere on disk or in memory, StreamSQL supports operations on streams of continuously changing data. The two application formats are functionally equivalent; in fact StreamBase Studio provides a utility that converts EventFlow sbapp files to StreamSQL ssql files.

SQL쿼리도 그림으로 처리하고싶으면 StreamSQL을 만들어…. StreamSQL은 기존 SQL이나 비슷해… 근데 쪼오금 다른점이 있지…

기존 SQL은 정리된 데이터를 대상으로 했지만 StreamSQL은 동적인 데이터를 대상으로 한다는점. 두 어플리케이션의 형식은 같다; 스트림베이스는 sbapp파일을 ssql파일로 변환하는 툴을 제공한다.

You can use any text editor to write StreamSQL applications. However, coding in StreamBase Studio offers several advantages:

StreamSQL을 메모장 같은 텍스트에디터로도 할 수 있는데… 바보가 아닌한…. 누가 그럴라고 ..편한이유 읽어볼 필요도 없을듯

  • The SB Authoring perspective includes a StreamSQL editor that facilitates writing StreamSQL queries. The editor’s Content Assist feature makes entering keywords easy, and StreamBase checks your syntax as you work. 키워드를 쉽게 찾아준데
  • StreamBase Studio views make it easy to combine StreamSQL applications and other StreamBase components. For example, you might want to reference a StreamSQL application within a StreamBase EventFlow application. Or, going the other way, you might embed existing StreamBase applications as modules within a StreamSQL query. 아이 편해

The following image shows the StreamSQL Editor in the SB Authoring perspective:

이 그림이 뭔지는 보면 알겠지…:

————————————————————————————————————————-

Applications As Modules 모듈화프로그래밍

As we mentioned earlier, an EventFlow or StreamSQL application can be used as a module in another application. For example, you can create an application that performs a common task, and include it in any number of other applications. This reuse can save time in both development and maintenance. Obviously, the functionality in the module must make sense for the application that uses it. For example, both must use compatible data.

내가 말했지? 이벤트 플로우나 스트림SQL이나 이런거 다 모듈로 만들어서 딴 어플에 쓸 수 있다고. 예를들어서 ………………………………예를들필요있나… 프로그램 해봤음 다 알잖아… 모듈화 코드재사용 객체지향

If you are working on an EventFlow application, StreamBase Studio lets you simply drag an existing application from the Package Explorer to the canvas of the EventFlow Editor. In the following figure, an StreamSQL module has been added:

StreamSQL을 만들고나서… 이렇게 긁어넣으면 이렇게 들어가는거야

If you are writing a StreamSQL application, you can include an existing application as a module.

StreamSQL어플리케이션 만들어놓은고… 화면에 긁어넣어.

 

————————————————————————————————————————-

Summary and Next Steps 다음장으로….

We have covered some of the key StreamBase components and terms, and the ways in which you can create applications. It’s time to start using the tool. You can learn much more about the product by following the remaining topics in this guide, where you will have an opportunity to build your first StreamBase application.

To get started, click Next.

StreamBase Partner Program스트림베이스파트너프로그램

http://www.streambase.com/partners-home.htm

The StreamBase Partner Program스트림베이스 파트너프로그램

Welcome to the StreamBase Partner Program. This guide describes the majorfeatures of the program including its scope, benefits, and the enrollmentprocess.The StreamBase Partner Program is designed for Technology Partners,Independent Software Vendors, System Integrators, Value-Added Resellers,and other providers of StreamBase driven solutions. It is a global program for helping our partners bring their real-time applications to market faster and moreeasily by using the StreamBase Complex Event Processing (CEP) platform.StreamBase is the market leader in CEP technology and we are dedicated to making our platform the strongest choice for building real-time event streamingapplications. Our partners depend on us to provide the best possible CEP platform and we depend on our partners to provide the best possible solutionsto end-users. That partnership is at the core of our business model and at thecore of the StreamBase Partner Program.

반갑다친구야. 여기서는 이 프로그램의 범위와 이익 가입방법이 써 있어. 스트림베이스 파트너 프로그램은 기술파트너, 독립 소프트웨어 벤더, System Integrators, 판매자, 그리고 다른 스트림베이서 운영 솔루션 공급자들을 상태로 하는거임. 이것은 파트너가 스트림베이스를 활용해서 실시간 어플 개발을 쉽고 빠르게 도와주는 글로벌 프로그램이야. 스트림베이스는 CEP기술에서 1등기업이고 우리는 이것을 더 발전시킬 생각.

…………….

아 어렵다.  읽기는 쉬운데 번역은 어려운 내용이네 진짜… 내용은 쉽고 안중요하니까.. 번역은 귀찮으니까… 넘어갈까…

Getting Started with StreamBase Studio

StreamBase Documentation >

http://docs.streambase.com/latest/index.jsp

Getting Started with StreamBase Studio

스트림베이스 시작하기

This guide is an introduction to StreamBase Studio. If you have installed StreamBase and are using Studio for the first time, this is a good place to start. After reading about essential concepts and terms, step through the Getting Started tutorial to learn basic skills through hands-on experience.

이 설명서는 스트림베이스를 소개합니다. 네가 스트림베이스를 설치하고 처음 쓴다면 여기부터 시작해. 기본적인 개념과 규정을 읽고나서 튜토리얼을 따라하면서 기초적인 사용법을 익히도록 해

The tutorial builds on the Cheat Sheet, also titled Getting Started, that is available from Studio’s Welcome page and in the Help menu. You will create the same application, but instead of entering individual tuples to test your running application, you will create a feed simulator that automatically enqueues tuples for you. The tutorial takes an estimated 15 to 20 minutes to complete.

튜토리얼은 가짜 시트에다가 만들건데 GettingStarted에도 있고…스트림베이스 Welcome의 Help 메뉴에서도 볼 수 있어.

너네는 똑같은 어플리케이션을 만들지만,… 튜플들은 다르다고?

너는 튜플을 자동적으로 enqueues시키는 프로그램을 만들거야

15분~20분정도면 따라할 수 있어~

Click the Next button at the top or bottom of the page to step through this guide. For maximum benefit, please follow the tutorial topics in sequence. Of course, feel free to back up (by clicking Previous) at any point to review an earlier topic. And you can always read suggested topics in StreamBase Help to explore more.

Next버튼하고 Previous버튼에 대해 설명을 하고 있네… 친절하기도 해라~

스트림베이스 프로그램 내의 헬프버튼을 통해서도 메뉴얼을 볼 수 있음여

Contents

Quick Facts: StreamBase Terms and Components
Launching StreamBase Studio
Creating Your First StreamBase Project
Writing Your First EventFlow Application
Creating Your First StreamSQL Application
Running Your First StreamBase Application
Creating a Feed Simulation
Sending Test Data to Your StreamBase Application