Dds the ideal_bus_for_event_processing_engines

  • Published on
    20-Aug-2015

  • View
    795

  • Download
    0

Embed Size (px)

Transcript

<ol><li> 1. Data-DistributionService (DDS)The Ideal Bus for CEP dds/2008-03-04 Gerardo Pardo-Castellote, Ph.D. Co-Chair Data-Distribution SIG CTO, Real-Time Innovations, Inc. 13 March 2008 </li><li> 2. Agenda Introduction Middleware and CEP The OMG DDS standard as a message bus for CEP Conclusion2 2007 Real-Time Innovations, Inc. </li><li> 3. History:DDS the Standard Data Distribution Service for Real-Time Systems Adopted in June 2003 Finalized in June 2004 Revised June 2005, June 2006 Joint submission (RTI, THALES, OIS) Specification of API for Data-Centric Publish-Subscribe in real-time distributed systems. Interoperability wire protocol Adopted in July 2006 Revised in July 2007 Multiple Implementations 4 commercial 3 open source Several more in-house3 2007 Real-Time Innovations, Inc. </li><li> 4. DDS Adoption Aerospace &amp; DefenseBoeingAWAKSNorthtropE2C Hawkeye Boeing Future Combat SystemsRaytheon SSDS Lockheed AEGIS Insitu Unmanned Air Vehicles4 </li><li> 5. DDS Adoption Transportation, Industrial WiTronix Train and vehicle Schneider Electric Tracking Industrial AutomationTokyo JapanTraffic Control Kuka RoboticsEU and USAir TrafficManagement Varian Medical Instruments5 </li><li> 6. Many others6 2007 Real-Time Innovations, Inc. </li><li> 7. DDS Mandates for Net-Centric Systems DISR (formerly JTA) DoD Information Technology Standards Registry US Navy Open Architecture FCS SOSCOE Future Combat System System of System Common Operating Environment NESI Net-centric Enterprise Solutions for Interoperability UK MOD &amp; NATO Advocating Open Systems7 2007 Real-Time Innovations, Inc. </li><li> 8. Middleware and CEP: The hose that feeds the engineInputs OutputsCEP Engine CEP Engine8 2007 Real-Time Innovations, Inc. </li><li> 9. Middleware and CEP: Scalability &amp; Load Balancing CEP EngineCEP EngineCEP Engine CEP Engine9 2007 Real-Time Innovations, Inc. </li><li> 10. Middleware and CEP: Integration &amp; InteroperabilityVendor 2Vendor 3Vendor 1Vendor 410 2007 Real-Time Innovations, Inc. </li><li> 11. Implementing the Global Event Bus/Data StreamsEventType1EventType2Global Event BusEventType2 ADAPTOR != MIDDLEWARE Multiple adaptor technologies will be needed but not all are created equal It is highly desirable to use a middleware where the event model and use- case requirements are handled as first-class citizens Event semantic model can be pushed into the middleware information11 model 2007 Real-Time Innovations, Inc. </li><li> 12. Standardizing the Adaptor/Transport NEEDED: Not so much a standard adaptor but: A standard event meta-model Relevant OMG standards: UML, MOF, OCL, A standard way to adapt to the existing Transports/MessageBusses &amp; information model Using the same Adaptor technology will not allow CEP engines to interoperate Relevant OMG standards: DDS, CORBA Notification ServiceCommunicate != Interoperate12 2007 Real-Time Innovations, Inc. </li><li> 13. Requirements on a CEP Message Bus Fit to the CEP information Model CEP engines need access to data regardless of its source CEP operates on structured data, not opaque messages Performance &amp; Scalability CEP systems are event-driven CEP operates on streaming data and has low-latency &amp; high-throughput requirements Configurability, QoS, Filtering &amp; Services CEP is deployed in critical systems that be robust to failures CEP weaves streams with different QoS requirements (updatesrates, priority) CEP often needs access to only a subset of the information13 2007 Real-Time Innovations, Inc. </li><li> 14. DDS: The best message bus for CEP? Fit to the CEP information Model DDS is Data-Centric, not message centric Beyond messaging DDS also does state management DDS Supports Topic-Based and Content-Based Subscriptions DDS built-in Window/Time selectors delivers *only* relevant data Performance &amp; Scalability DDS offers notification-based APIs DDS has the best performance (latency, throughput, jitter, scalability) of any standard middleware technology Configurability, QoS, Filtering &amp; Services DDS Publish-Subscribe model delivers wherever needed DDS has rich QoS to support real-time event-driven systems (latencybudget, priority, etc.) DDS Built-in Fault-Tolerance Mechanism allow for redundant sources ofdata DDS built-in services Persistence and Historical Data Services ensuresrelevant data is never lost14 2007 Real-Time Innovations, Inc. </li><li> 15. #1 DDS Data-Centric Model Global Data Space generalizes Subject-Based Addressing Data objects addressed by DomainId, Topic and Key Domains provide a level of isolation Topic groups homogeneous subjects (same data-type &amp; meaning) Key is a generalization of subject Key can be any set of fields, not limited to a x.y.z formatted string Data Structure is known by the middleware Data Reader Data WriterTopic Data Reader Data WriterData Reader Data Writer15 2007 Real-Time Innovations, Inc. </li><li> 16. #1 DDS Data-Centric Model Global Data Space generalizes Subject-Based Addressing Data objects addressed by DomainId, Topic and Key Domains provide a level of isolation Topic groups homogeneous subjects (same data-type &amp; meaning) Key is a generalization of subject Key can be any set of fields, not limited to a x.y.z formatted string Data Structure is known by the middleware Data Reader Data WriterKey (subject) Data Reader Data WriterData Reader Data Writer16 2007 Real-Time Innovations, Inc. </li><li> 17. #1 DDS Data-Centric Model Global Data Space generalizes Subject-Based Addressing Data objects addressed by DomainId, Topic and Key Domains provide a level of isolation Topic groups homogeneous subjects (same data-type &amp; meaning) Key is a generalization of subject Key can be any set of fields, not limited to a x.y.z formatted string Data Structure is known by the middleware Data Reader Data WriterData Object Data Reader Data WriterData Reader Data Writer17 2007 Real-Time Innovations, Inc. </li><li> 18. Subscriptions: By Topic, Subject, ContentTopic: Market DataKey FieldsOther Fields PayloadFieldSource SymbolType Exchange Volume Bid Ask Value* ** * * Topic: Order EntryKey FieldsOther Fields FieldSymbol TypeExchange OrderNumber OrderKind StopLimit Value**NYSE*Subject Filter (for a Reader)Topic: Market DataKey Fields Other Fields FieldSource SymbolType Exchange Payload Value REUTERS *EQ NYSEVolume &gt; x, Ask &lt; y18 2007 Payload Filter (for a Reader) Subject Filter (for a Reader) Real-Time Innovations, Inc. </li><li> 19. DDS communications model Data DomainData DomainNewWriter ParticipantReaderParticipantAlarmGot newAlarm subscriber! dataOfferedOffered Listener QoS Listener QoSParticipants scope the global data space (domain)Topics define the data-objects (collections of subjects)Writers publish data on TopicsReaders subscribe to data on TopicsQoS Policies are used configure the systemListeners are used to notify the application of events19 2007 Real-Time Innovations, Inc. </li><li> 20. Demo: Concepts Start demoDisplay Area:Shows state of objects Topics Square, Circle, Triangle Attributes Data types (schemas) Shape (color, x, y, size)Color is instance Key AttributesShape &amp; color used for key QoS Deadline, Liveliness Reliability, Durability History, Partition Control Area: Ownership20 Allows selection of objects and Real-Time Innovations, Inc. 2007 QoS </li><li> 21. QoS: Quality of Service Standardized Middleware QoS semantics QoS PolicyQoS Policy DURABILITYUSER DATA HISTORY (per subject) TOPIC DATA READER DATA LIFECYCLE GROUP DATA WRITER DATA LIFECYCLE PARTITION LIFESPANPRESENTATION ENTITY FACTORYDESTINATION ORDER RESOURCE LIMITS OWNERSHIP RELIABILITY OWNERSHIP STRENGTH TIME BASED FILTER LIVELINESS DEADLINELATENCY BUDGET CONTENT FILTERS TRANSPORT PRIORITY21 2007 Real-Time Innovations, Inc. </li><li> 22. Demo: Quality of Service (QoS)Start demo Writers and readers state Their needsTopics Square, Circle, Triangle AttributesData types (schemas) Shape (color, x, y, size) Color is instance Key Attributes Shape &amp; color used for keyQoS Deadline, Liveliness Reliability, Durability History, Partition Ownership RTI DDS delivers22 2007 Real-Time Innovations, Inc. </li><li> 23. DDS: The best message bus for CEP? #1 Fit to the CEP information Model DDS is Data-Centric, not message centric DDS Supports Topic-Based and Content-Based Subscriptions DDS built-in Window/Time selectors delivers *only* relevant data #2 Performance &amp; Scalability DDS offers notification-based APIs DDS has the best performance (latency, throughput, jitter, scalability) of any standard middleware technology #3 Configurability, QoS, Filtering &amp; Services DDS Publish-Subscribe model delivers wherever needed DDS has rich QoS to support real-time event-driven systems (latencybudget, priority, etc.) DDS Fault-Tolerance Mechanism allows for redundant sources of data DDS built-in services Persistence and Historical Data Services ensuresrelevant data is never lost23 2007 Real-Time Innovations, Inc. </li><li> 24. Data-Distribution and Real-TimeMessaging Technologies and StandardsWeb Services JavaRTSJ (soft RT) RTSJ (hard RT) Java/RMI Java/JMS CORBA RT CORBAData Distribution Service / DDS MPI Non-real-timeSoft real-timeHard real-time Extreme real-time24 2007 Real-Time Innovations, Inc. Adapted from NSWC-DD OA Documentation </li><li> 25. Latency (Linear Scale) IBM HS20 blades Dual 2.8 GHz Xeon, 1 GB RAM Gigabit EthernetDDS/JMS/Notification Service Comparison - Latency25002000DDS JMSNotification Service150010005000 163264128 256 512 10242048 4096 8192 16384 32768 65536 Message Size (bytes)Adapted from Vanderbilt presentation at July 2006 OMG Workshop on RT Systems25 2007 Real-Time Innovations, Inc. </li><li> 26. Jitter (Linear Scale)DDS/CORBA Notification Service Comparison - Jitter100 DDS/JMS/CORBA Notification Service Comparison - Jitter Standard Deviation (usecs) 8020001800DDS JMS Notification service 60 Standard Deviation (usecs)16001400 40DDSJMSNotification service12001000 20 800 6000 400 163264128 256512 1024 20484096 8192 16384 32768 65536 200Message Size (bytes) 01632 641282565121024 2048 409681921638432768 65536 Message Size (bytes) Source: Vanderbilt presentation at July 2006 OMG Workshop on RT Systems26 2007 Real-Time Innovations, Inc. </li><li> 27. DDS : Low Latency and Jitter Latency and Jitter on Unloaded Network400 Latency (microseconds)350300 Maximum250 99.99%20099%150Median Minimum10050 032 64 128 256 512 1024 2048 4096 8192Message/Data Size (bytes)Reliable, ordered delivery overGigabit Ethernet between 2.0 GHz Opteron processorsrunning 32-bit Red Hat Enterprise Linux 4.027 2007 Real-Time Innovations, Inc. </li><li> 28. DDS: Low Overhead Enables HighThroughput70,000100090060,00080050,000700 Megabits per Second Updates per Second60040,00050030,00040020,00030020010,0001000 0 16 32 641282565121024 2048 4096 8192 16384 32768 Message/Data Size Reliable, ordered delivery over Gigabit Ethernet between 2.0 GHz Opteron processors running 32-bit Red Hat Enterprise Linux 4.028 2007 Real-Time Innovations, Inc. </li><li> 29. DDS: Scalable Performance (ConfirmedReliability) 60,000 50,000 Point-to-Point Updates per Second 40,000 1-11-101-24 30,000 20,000 10,000 016 3264 128 256 5121024Message/Data Size (bytes - without batching) Reliable, ordered delivery over Gigabit Ethernet between 2.0 GHz Opteron processors29 running 32-bit Red Hat Enterprise Linux 4.0 2007 Real-Time Innovations, Inc. </li><li> 30. DDS operates without brokers DDS Broker-based Middleware No Brokers =&gt; Better Performance &amp; Determinism + Increased Robustness30 2007 Real-Time Innovations, Inc. </li><li> 31. DDS: The best message bus for CEP? #1 Fit to the CEP information Model DDS is Data-Centric, not message centric DDS Supports Topic-Based and Content-Based Subscriptions DDS built-in Window/Time selectors delivers *only* relevant data #2 Performance &amp; Scalability DDS offers notification-based APIs DDS has the best performance (latency, throughput, jitter, scalability) of any standard middleware technology #3 Configurability, QoS, Filtering &amp; Services DDS Publish-Subscribe model delivers wherever needed DDS has rich QoS to support real-time event-driven systems (latencybudget, priority, etc.) DDS Fault-Tolerance Mechanism allows for redundant sources of data DDS built-in services Persistence and Historical Data Services ensuresrelevant data is never lost31 2007 Real-Time Innovations, Inc. </li><li> 32. #3 QoS and Powerful Services Redundancy &amp; QoS PolicyQoS Policy Failover DURABILITYUSER DATA Persistent DataHISTORY (per subject) TOPIC DATA Last value cache READER DATA GROUP DATALIFECYCLE Historical cache WRITER DATA PARTITIONLIFECYCLELIFESPANPRESENTATIONENTITY FACTORYDESTINATION ORDERRESOURCE LIMITS OWNERSHIPRELIABILITY OWNERSHIP STRENGTHTIME BASED FILTER LIVELINESSDEADLINELATENCY BUDGETCONTENT FILTERS TRANSPORT PRIORITY32 2007 Real-Time Innovations, Inc. </li><li> 33. Ownership and High Availability Producer / Writer I1 Prim strength=10ary Topic T1 Producer / Writer I1 Backupstrength=5I2 Primary I1 I2 Producer / Writer I2 Backupstrength=1Owner determined per subjectOnly extant writer with highest strength can publish a subject (ortopic for non-keyed topics)Automatic failover when highest strength writer: Loses liveliness Misses a deadline Stops writing the subjectShared Ownership allows any writer to update the subject33 2007 Real-Time Innovations, Inc. </li><li> 34. Data Persistence A standalone service that persists data outside of the context of a DataWriter DataData WriterDataWriterReaderData Global ReaderData SpacePersistence PersistenceService Service PermanentPermanentStorageStorage34 2007 Real-Time Innovations, Inc. </li><li> 35. Conclusion To get the best performance from your CEP you need the best message bus DDS is a natural fit for CEP: The Data-Centric Pub-Sub information model is an excellent match to the CEP structured event model DDS supports many of the CEP use-cases and QoS as first-class citizens DDS offers the best performance and latency of any middleware standard35 2007 Real-Time Innovations, Inc.</li></ol>