аЯрЁБс>ўџ 02ўџџџ/џџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџьЅСq Пœbjbjt+t+ ",AAœџџџџџџ]&&&&&&&::::: F:[ 2ffffffff " " " " " " $ є ўF &fffffF ц&&fffцццfF&f&f ::&&&&f цHц.:ъ,&& fZ №FЇЧИЦ::Ќ: Index/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion Description Routine Redundant Operations: Data stored on an ISV Node is also found on two other ISV Nodes for both redundancy and processing efficiency. As an ISV Node processes data it replicates the pertinent data on it’s two companion nodes over the Communications Backchannel(s) and also stores it to the SAN. A heartbeat/synchronization function between these three ISV Nodes over this same Backchannel is used to provide an “ISV healthy” indication and synchronization among the set of three. Note that though each user’s data is replicated on three machines, which three machines varies for each user. As a result, machines are not necessarily companioned with any particular other machines—just the per-user data. If a machine goes down, two machines might have to pick up the load for one user metadata index, but another two machines would pick up the load for another user metadata index. Failed or “Turned” ISV Node: If an ISV Node fails to complete a data transaction or a heartbeat sequence the other two ISVs interact to replace its functionality. See note above. Perhaps we should be more vague and just say, “other ISV nodes interact to replace the down node’s functionality.” There are two phases that a failure can be identified in: Data Communications: If an ISV fails or is shut down/reset in the middle of transmitting data an attempt is made to reestablish communications by one or all of the remaining nodes. Do you mean transmitting data to/from the SAN, or to/from the MSG machine? My take on this was to see if we could get some breathing room by promising to never lose index/profile data, but allow loss of search/view transactions. I have no feedback yet from Bloomberg if this is ok, but if it’s ok for a downed node’s search/view transactions to be lost, perhaps that would help accelerate our implementation. If communications are reestablished the interrupted node is asked to retransmit the data. If this is a recurring event at some point an Exception Message is sent to the SNMP engine indicating a malfunctioning/misbehaving ISV Node by one or more ISV Nodes in the set. If communications are not reestablished the remaining two nodes seek out another node willing to take on an additional load and then redistribute the failed node’s workload to the new node. An Exception Message is sent back to the VAPI by one or both of the remaining set. This implies a certain level of cooperation between the remaining nodes to accomplish this task. Heartbeat Sequence: A failure of an ISV Node to initiate or complete a Heartbeat Sequence in a timely manner will instigate a series of retry attempts: If it turns out to be okay for us to lose search/view transactions (but not index transactions), then we may not need to use a heartbeat for our implementation. Submitted index requests would simply be stored on the SAN, and later any (other) machine could scan for pending work and complete the index transaction. In this scenario, the heartbeat might be useful in a scenario like this: if a machine locks a collection (user metadata index) for an indexing operation, but has not unlocked the collection within a timeout period, then another machine might notice the incomplete work and send a ping to the “down” machine. If no heartbeat is returned, then the lock and in-progress index work is cancelled, and the index transaction is resubmitted. If the heartbeat is returned, then we do something to establish that the machine really is working towards completion of the unfinished indexing task. If a Heartbeat is reestablished the node is asked to rejoin the set and complete any unfinished tasks it has pending. If this is a recurring event at some point an Exception Message is sent to the SNMP engine indicating a malfunctioning/misbehaving ISV Node by one or more ISV Nodes in the set. This implies a certain level of cooperation between the remaining nodes to accomplish this task. If a Heartbeat is not reestablished the remaining two nodes seek out another node willing to take on an additional load and then redistribute the failed node’s workload to the new node. An Exception Message is sent to the SNMP engine indicating a malfunctioning/misbehaving ISV Node by one or more ISV Nodes in the set. This implies a certain level of cooperation between the remaining nodes to accomplish this task. Fail-Back/Dynamic Expansion: Since Fail-Back and Dynamic Expansion have the same functional implications we will treat them here in the same scenario. Fail-Back is defined as the action required when an ISV Node is replaced after failure or shut down/reset. Dynamic Expansion is defined as the addition of a new ISV Node to the array to increase performance and/or capacity. As you will see the resulting actions on the part of the ISV Node Array members are the same for both scenarios: Status Quo Situation: The ISV Array is in Status Quo or has made accommodations for a lost element by dividing up the element’s workload among two or three of the remaining ISV Nodes. In either case we are adding a “new” resource to the array. There should never be less than two nodes responsible for user data. Resource Add Scenario: A new ISV Node is plugged into the network switches and SAN front-end and powered up. The new ISV Node broadcasts its presence and does a discovery of its environment. At the same time the other attached nodes, MSG Machines, and the SAN front-end see the new arrival and make a note of it in their routing tables. The new ISV Node then searches out another ISV set that could use some offloading and negotiates a sharing of the user load. Incomplete set of two nodes available: These two other ISV Nodes download their user tables and the new node takes its position as a member of the new set. Heartbeat and data communications resume as normal. Only full triad sets available: If all of the ISV Nodes in the array are members of a complete set (3 ISVs) then one set will relinquish a node and the new node will pair with it for a set of 2. The set relinquishing the node will transfer ownership of a certain number of users to the exiting node and copy those to the new node also. Heartbeat and data communications resume as normal. RTsƒŽŸбяuњ5 I œ%Pl:OtŠDj5œћљљїїљїљљїљљљљљB*55CJ RSTs=абя4 5 ъ …  ‘ œ%­ЎOPn9ќќќњњњієњњіі№ющхщ№№№щхщњњі„p & F„ „а$RSTs=абя4 5 ъ …  ‘ œ%­ЎOPn9:stCD›œ§ћјјјјѕѕѕѕ  #9:stCD›œ§јєј§ј§јє„а & F 1hАа/ Ар=!А"А# $ %А [8@ёџ8 NormalCJ_HaJmH sH tH <A@ђџЁ< Default Paragraph Font<C`ђ<Body Text Indent„аB*@R`@Body Text Indent 2„ B*œ , џџџџœ9œœowРШэіI Q #žадВГ  % žџџBrent D. MillerBC:\DOCUME~1\BMILLE~1.BMI\LOCALS~1\Temp\AutoRecovery save of fb.asdBrent D. Miller=\\192.168.2.38\bmiller\work\veroquest\bloomberg\060122\fb.doc^V ' АЌCџџџџџџџџџ(РPВъœ=џџџџџџџџџh„ „˜ўЦ ^„ `„˜ўOJQJo(‡hˆHЗ№h„p„˜ўЦp^„p`„˜ўOJQJ^Jo(‡hˆHoh„@ „˜ўЦ@ ^„@ `„˜ўOJQJo(‡hˆHЇ№h„„˜ўЦ^„`„˜ўOJQJo(‡hˆHЗ№h„р„˜ўЦр^„р`„˜ўOJQJ^Jo(‡hˆHoh„А„˜ўЦА^„А`„˜ўOJQJo(‡hˆHЇ№h„€„˜ўЦ€^„€`„˜ўOJQJo(‡hˆHЗ№h„P„˜ўЦP^„P`„˜ўOJQJ^Jo(‡hˆHoh„ „˜ўЦ ^„ `„˜ўOJQJo(‡hˆHЇ№h„@ „˜ўЦ@ ^„@ `„˜ўOJQJo(‡hˆHЗ№h„„˜ўЦ^„`„˜ўOJQJ^Jo(‡hˆHoh„р„˜ўЦр^„р`„˜ўOJQJo(‡hˆHЇ№h„А„˜ўЦА^„А`„˜ўOJQJo(‡hˆHЗ№h„€„˜ўЦ€^„€`„˜ўOJQJ^Jo(‡hˆHoh„P„˜ўЦP^„P`„˜ўOJQJo(‡hˆHЇ№h„ „˜ўЦ ^„ `„˜ўOJQJo(‡hˆHЗ№h„№„˜ўЦ№^„№`„˜ўOJQJ^Jo(‡hˆHoh„Р!„˜ўЦР!^„Р!`„˜ўOJQJo(‡hˆHЇ№(РP^V 'џџџџџџџџџџџ@€$$dЭШ$ž œА@G‡z€џTimes New Roman5€Symbol3& ‡z€џArial?5 ‡z€џCourier New;€Wingdings"qˆ№аh‹ГЁУДЁƒJ +—Ф!№ ДД20ъ[џџMIndex/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion ScenariosPatti and MikeBrent D. Millerўџр…ŸђљOhЋ‘+'Гй0Ф˜№ќ ,@ Xd € Œ ˜ЄЌДМфNIndex/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion ScenarioslndePatti and Mikeeattatt Normal.dotiBrent D. Miller16nMicrosoft Word 8.0I@вьL@"RіЦ@њ СИЦJўџеЭеœ.“—+,љЎDеЭеœ.“—+,љЎx4 hp|„Œ” œЄЌД М щ§?+ ъ1 NIndex/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion Scenarios TitleŒ@НХ%Eq _PID_GUID_AdHocReviewCycleID_EmailSubject _AuthorEmail_AuthorEmailDisplayNameщ§AN{91129729-0AC2-45F8-A7EA-D6A1F8666C06}~љFirst cut @ F/O, F/Bme"цНЂцЉЎч•Џч•Вч€цЂцБЇц‰ЏцБЁцИЎч‘Ѕ,  Murphy-Steen€ц ўџџџўџџџ !"#$%&ўџџџ()*+,-.ўџџџ§џџџ1ўџџџўџџџўџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџRoot Entryџџџџџџџџ РFп№[ЦакЋЧИЦ3€1TableџџџџџџџџџџџџWordDocumentџџџџџџџџ",SummaryInformation(џџџџDocumentSummaryInformation8џџџџџџџџџџџџ'CompObjџџџџjObjectPoolџџџџџџџџџџџџакЋЧИЦакЋЧИЦџџџџџџџџџџџџўџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџўџ џџџџ РFMicrosoft Word Document MSWordDocWord.Document.8є9Вq