Integration Server and Java

[PhilLeary]

Flow services are not converted to Java code. System methods run to read the flow service and execute the commands within it, but at no time is the flow "code" converted to real Java.

The argument for Java versus Flow should perhaps be conducted at two levels: developer and project manager.

The developer will likely always argue for Java - there are third-party tools such as Eclipse that make it easy to code and debug, there is excellent documentation and a supportive community, it's fun to program in, the runtime performance is "better" within the webMethods suite, and etc, etc.

The project manager who is concerned with time to market, return on investment, and mid-term maintainability will likely argue for Flow. It is easier (and less costly - in dollars and project time) to train someone on Flow than Java - so development and maintenance can be performed with a larger pool of "programmers".

But the bottom line is this:
I have worked with Java since .9beta and Flow since 1999. Like any good developer, I choose the appropriate tool for the job, looking at both the immediate task and its impact within the lifecycle of the greater application/initiative. Sometimes that is Java, sometimes that is Flow.

[reamon]

Originally Posted by pmarsteau

Flow services are interpreted and at the bottom line converted to java code.

No, they are not. As Phil Leary points out in his reply lower in the thread, FLOW is more or less interpreted on the fly.

Originally Posted by pmarsteau

So basically coding in Java will always be faster than coding in flow. My belief and my experience is coding in flow is suited for developers who do not have java background expertise.

I certainly respect your experience and opinion. I have no doubt about your effectiveness within IS. However, whether something can be done faster in Java or FLOW will almost certainly depend on the task at hand. Time and time again I've seen solutions developed in FLOW an order of magnitude faster than the equivalent in Java. I've also seen the reverse, but that's much more rare.

Originally Posted by pmarsteau

Any real java programmer will always code faster a service in java than in flow (that at the end result in java code).

This is indeed the entire point of the thread. If you're a Java programmer, and not comfortable with FLOW, you'll certainly be faster coding Java services. But you'll be doing so at the expense of debugging, maintenance, etc. The primary programming language of IS is FLOW, not Java. Put a capable FLOW programmer next to a Java programmer, and they will compare favorably in terms of productivity. For some tasks, the FLOW programmer will easily finish first. For some other tasks, the Java programmer will finish first.

Originally Posted by pmarsteau

About readiness and debugging facilities, we do all of our coding, testing and debugging directly within eclipse IDE.

That's certainly doable. I assume you make heavy use of the Java API and the Service.invoke method? I assume that a good part of the code is simply building pipelines and retrieving pipeline values. This begs the question: Why are you using IS? Why not WebSphere or WebLogic, environments that are geared for Java programming?

Originally Posted by pmarsteau

Debugging is easier (and faster!) using eclipse using Developer IDE.

This is arguable. I have never encountered a case where I thought debugging was hard, or slow. I've done a fair amount of raw Java programming and I'd argue that the amount of effort and the speed are about the same.

Originally Posted by pmarsteau

Code is faster inline documented in Java code than using the Developer.

Speed of code execution is but one measure of a solution. Most of the time, this is immaterial to the overall solution and other factors are more important.

Originally Posted by pmarsteau

You can badly code Java or code it the correct way, as you can "code" badly flow services or in the correct way.

Agreed.

Originally Posted by pmarsteau

Handling exceptions in Java is way easier using Java than using TRY CATCH blocks in Flow (that are anyway handling underlying Java ServiceException).

Agreed. Exception management within FLOW is quite awful.

Originally Posted by pmarsteau

In what I read in this thread I can buy no real arguments that would convince me to use Flow rather than Java, except if you cannot or do not know how to code Java efficiently.

That's one of the primary reasons for the success of IS--you don't need to be a Java programmer. It's definitely a plus if you are but if you're not, you can still be quite effective.

Do your solutions use FLOW services at all? Or is everything mostly Java services? Do you use the built-in services extensively? How much FLOW background do you have?

This thread was prompted by post such as "how can I do a Java service that will parse XML" and other posts that plainly show a lack of understanding of the "sweet spot" of IS and the basics of FLOW and the built-in services. It wasn't to say never use Java. The key point was "know the tools you're working with" and which to use when. My opinion is too many default to using Java services when using FLOW services would be a smarter choice.

The other extreme you seem to follow, do it as Java services first, rarely as FLOW services, seems odd to me but I can see how such an approach has a certain appeal.

It would be interesting to see the two approaches compared head-to-head. Pit a developer proficient in FLOW (primary) and Java (secondary) against a developer proficient in Java (primary) and FLOW (secondary) and see who is able to build a "better" solution based on several criteria.

[pmarsteau]

Originally Posted by reamon

No, they are not. As Phil Leary points out in his reply lower in the thread, FLOW is more or less interpreted on the fly.

Each flow step is interpreted and as for consequence to invoke a Java method that perform the given step action. The most evident about this point is the underlying JVM can only understand Java, and does not know nothing about properietary wM Flow language and commands. Arguing that each step is not converted to Java invocation just make no sense.

Originally Posted by reamon

I certainly respect your experience and opinion. I have no doubt about your effectiveness within IS. However, whether something can be done faster in Java or FLOW will almost certainly depend on the task at hand. Time and time again I've seen solutions developed in FLOW an order of magnitude faster than the equivalent in Java. I've also seen the reverse, but that's much more rare.
This is indeed the entire point of the thread. If you're a Java programmer, and not comfortable with FLOW, you'll certainly be faster coding Java services. But you'll be doing so at the expense of debugging, maintenance, etc. The primary programming language of IS is FLOW, not Java. Put a capable FLOW programmer next to a Java programmer, and they will compare favorably in terms of productivity. For some tasks, the FLOW programmer will easily finish first. For some other tasks, the Java programmer will finish first.

I agree that a good IS Flow developer will be very fast in terms of coding a specific feature. I coded flow exclusively for about 3 years before discovering how easy it was to code within eclipse in Java. I was considered pretty knowledgeable and competent in writing efficient and reusable flow services.

However, the time to build an if-then-else or a switch-case sequence in FLOW will always be far longer than coding it in Java, even if you are not a Java master. Writing (or actually typing) few lines of Java code is for me way faster than dragging and dropping the different steps and populating their properties to specify the if/case conditions. Now think about regex in Flow vs. in Java, and you will probably agree the easiness of Java to refer regex groups in Java vs. in Flow... Or think about the CPU time to concatenate String in Flow, compared to using StringBuffer in Java, and you will be remain perplex about the time to execute in Flow vs. Java (about 400% factor, we measured it actually). Or think about adding 2 ints in Flow rather than typing a + b in Java, you will get more and more frustrated. Maybe its just a matter of badly designed IDE, but I still convinced about the benefits for me to code in Java rather than Flow (as long as you do not code Java directly in Developer IDE, which is almost useless).

Originally Posted by reamon

That's certainly doable. I assume you make heavy use of the Java API and the Service.invoke method? I assume that a good part of the code is simply building pipelines and retrieving pipeline values. This begs the question: Why are you using IS? Why not WebSphere or WebLogic, environments that are geared for Java programming?

That is however a good point. The bad using Java is to get away of the pipeline mentality, except where absolutely required. Our code is now almost exclusively Java only, using standards -mostly open-source- libraries (as webMethods itself does internally). Because some features are wM only, we still have to invoke some Service.invoke methods, however wrapped in a framework that makes our code "pipeline"-free (actually IData free). We keep these invocations as rare as possible, and pretty rarely needs them. And our strategy is indeed to be as decoupled as possible from wM code, in order to not be bound to them eternally. IS was implanted before I joined the company, and there were lots of debate about the ROI it brings in term of maintainibility, etc. Our inhouse consultants use to code FLOW only, mostly by lack of Java knowledge, and the biggest issues in production are caused by the Flow services of their packages. Because they badly use memory, performance, and reusability.

Originally Posted by reamon

This is arguable. I have never encountered a case where I thought debugging was hard, or slow. I've done a fair amount of raw Java programming and I'd argue that the amount of effort and the speed are about the same.

Developer IDE does not provide the same amount of debugging features than Eclipse does in term of watching expression, evaluating/modifying variables, catching exceptions, etc. Developer IDE is also very slow compared to Eclipse.

Originally Posted by reamon

Speed of code execution is but one measure of a solution. Most of the time, this is immaterial to the overall solution and other factors are more important.

Agreed. But memory (large file handling), IO and CPU usage are fairly important factors of stability in a production environment. And all these are pretty badly handled when using flow services.

Originally Posted by reamon

That's one of the primary reasons for the success of IS--you don't need to be a Java programmer. It's definitely a plus if you are but if you're not, you can still be quite effective.

Agreed. Being able to code Java in the IS brings definitively more than just coding Flow. Because all are not, IS made the right choice to scope more potential users than just Java developers.

Originally Posted by reamon

Do your solutions use FLOW services at all? Or is everything mostly Java services? Do you use the built-in services extensively? How much FLOW background do you have?

Mostly Java, because you lose the boosted performance when dealing with flow with Java code. We use either built-in service provided at the Java API of Broker and IS, or we use open-source mechanism to achieve the same features as those provided only at a flow level by webMethods. Most of the built-in services are in fact coded in Java, therefore relying on JRE available classes and methods. I have about 4 years of experience of FLOW, and use almost exclusively Eclipse and Java since about more than 1 year.

Bottom line, as we already agreed upon, there are good and bad developers.

To take it yet a little further, I tend to think that if you take a developer that badly codes in Java and badly codes in Flow, it may be easier to troubleshoot its bad flow service. If you take a developer that both masters Flow and a Java code, I tend to think that he will easier troubleshoot and read Java services than Flow services.

Nevertheless, do not take me wrong. I feel IS is a good product, cover most of the integration needs a company may need, and because it is based on Java, which is very extensive. That is a big plus. We use a lot of other products of webMethods other than IS and we are not about to challenge this product for another one - yet.

[reamon]

Originally Posted by pmarsteau

Each flow step is interpreted and as for consequence to invoke a Java method that perform the given step action. The most evident about this point is the underlying JVM can only understand Java, and does not know nothing about properietary wM Flow language and commands. Arguing that each step is not converted to Java invocation just make no sense.

Certainly executing FLOW services causes methods within various Java classes to be executed. But your statement that "FLOW is converted to Java" makes it sound like FLOW is translated to Java source/byte code in-place and then executed. You may not have meant to imply that but that's what came across.

Taking the analogy to the next level, FLOW is "converted" to Java byte code, which in turn is "converted" to machine code. So ultimately we should all be writing in assembler?

Originally Posted by pmarsteau

But memory (large file handling), IO and CPU usage are fairly important factors of stability in a production environment. And all these are pretty badly handled when using flow services.

The run-time performance of any environment is only as good as the administrators and developers. FLOW and IS, in and of themselves, all handle large files, IO and CPU just fine. Certainly it's easy to abuse these things though. Not managing memory appropriately or not closing files or not profiling to see where bottlenecks are killers in any run-time environment, not just IS. As you mentioned at the outset, sloppy programming will hurt any environment. I assume you've seen good and bad examples of both IS and Java app server environments.

All of your points are well taken. I won't counter any more of your points though because ultimately, based on what you've stated, I don't think you should be using IS in any way. You (when I say you, I mean your group/company) clearly do not want to do things the "IS way" and should therefore move on to tools that help you do things the way you want--which you have at least partly done already. Coding most everything in Java and using IS as a run-time environment for that code seems to me to be a serious mismatch.

It's not clear, from the exchange so far, the value IS actually brings to your environment. If you want to do development in Java, WS and WL (or your favorite open-source run-time) are far better environment choices.

[reamon]

Amen brother.

[pmarsteau]

Thanks for sharing your thoughts. There is always something good in challenging an architecture: I truly appreciate that.

We currently use Monitor and Modeler to model BPM for our integration needs, and built a kind of layer on top of IS to let our consultant implement customer specific needs by only configuring this (framework) layer using descriptive XML and XSL files.

As opposite to earlier developments, where each customer needs was coded in a separate package, involving more RAM usage, and introducing more often than less bugs or preventing efficient re-usability, they now only need to focus on mapping and transformations; no need to worry about how to code the request files retrieval or the result files delivery (i.e. to a dropbox, email, http, etc), no need to worry about how to make sure large file is properly handled, no need to worry about how to log errors in a consistent way, enabling monitoring and alerting of integration support or production teams on errors or failed transactions, etc... All these are taken care by our framework.

Adding a customer is almost as simple as adding some configuration XML files, routing XML configuration files, and adding/coding some XSL files for the diverse required transformations. Nonetheless, the framework already comes with standard integration processes (i.e. CSV import or CSV export of our business application entities), so that for some customers, no custom XSL files is even required as the customer gets the CSV files we sell him as part of our offering directly from our framework layer. When a customer specific needs happen to not be yet supported by the layer, there is 2 ways:
- either we refuse it to the customer
- or the consultant code a separate package and immediately request the DEV to implement the missing feature, so this one get available for any potential new customer in the future. A migration is then planned to remove the custom package and use the feature out of the framework, once available.

Because all this is coded in Java, and indeed reusable in another EAI, we cover the potential switch to another product, as long as this one would offer the same kind of features we use from webMethods: web server (TomCat), JVM (IS), MoM (Broker), BPM (Modeler/Monitor), and File Repository/DB for logging purposes. Note that we completely removed the usage of TN, because this was not suited to our needs and we needed some additional protocols that were not yet covered by the TN (i.e. SFTP is not supported on 6.1, authentication others than BASIC, or when using SAML in SOAP documents). We also use the SOAP interface of the IS so that our internal applications can use the IS as a web services provider.

Hope this helps you see what kind of usage we make from the IS: alone the Monitor to recode in a standalone Java application would be a very challenging aspect of getting away from IS. That's in part why we stick by it. By using a more Java approach of any custom code we create just covers our butt if we intend to switch for another EAI/SOA system in the future. Not clear what's wrong with this strategy.

[PhilLeary]

Your argument now seems to be that Java is your approach because of the potential reusability in another EAI suite. And you go further to state that as long as another EAI offered the same product categories (web server, JVM, MoM, BPM), that you could easily move to that other suite.

I understand from your email you've done some reasonably complex things in webMethods, but I'm confused by your argument that you could easily move those things to another EAI by virtue of having used Java.

That statement implies that none of your Java utilizes the webMethods API, all of your Models are pure BPEL, and your Broker messages are JMS (rather than the proprietary Broker format).

If this is not true, then your efforts are not reusable in another EAI.

If it is true, then I'm with Rob - it's not clear what value the IS brings to your environment.

[reamon]

The discussions are indeed always fun and a good chance to learn and appreciate other approaches and other points of view. Thanks for taking the time to discuss all this.

Question: Is your framework code hosted on IS? Or is it hosted elsewhere?

If on IS, much of what you describe about your framework sounds exactly like the frameworks I've either helped create or use at virtually every client I've worked at. All have the same goal--isolate the things that differ in each integration (mapping, etc.) and share the things that are common (routing, large file handling, partner management, logging, monitoring, alerts, etc.) The main difference between what you've described and what I'm describing is the implementation--you've chosen to use Java primarily while my past projects have primarily used FLOW services with a smattering of Java mixed in as needed.

There is nothing wrong at all with the custom Java strategy. It's one adopted by many. My initial reaction, though, is that you should go all the way with it. No need to stay tied to IS in any way.

Carve the Modeler/Monitor stuff off to an isolated service area--host just the models on IS and move everything else to WS, WLS or whatever. Modeler/Monitor, after all, used to be a separate entity entirely and was only recently folded into the IS environment. It doesn't seem unreasonable to create a "BPM server" that only runs models and only connects to Broker. If you do that, then your Java-based integration framework could be free of any and all IS ties. I think IS support of Tomcat has gone away, but I may be mistaken.

A custom integration environment, where you assemble components such as you listed, is definitely a reasonable approach. What gives me concern are hybrid approaches, such as running IS but then having most of the work be done by custom Java code. IS is not a Java app server. You get virtually none of the benefits that IS provides but you're paying for the license. The value in a tool like IS is not having to invent/develop much of the pieces. If you develop those pieces anyway, then why license the tool? My concern lies with your use of IS primarily--as a Java app server, it is not well suited since it is an integration server which has a much different focus (as you well know). If you're building most of the integration stuff yourself, the use of IS as a run-time (BPM is a separate issue) is questionable.

I understand that IS was inherited and so a transition period where there is a mix of both approaches is inevitable.

The path your environment has taken isn't surprising. I reiterate that a custom Java approach is absolutely reasonable. But I think you got there for the very reason that this thread was created--you ran into a road-block (couldn't do something in TN, or some set of stuff in IS didn't do quite what you wanted) and because of the Java background, you knew how to do it there. So that's what you did, instead of figuring out how to extend the tools you already had. None of the items you listed are insurmountable issues (custom TN delivery services, certificates, etc., I think Mark Carlson has created a SAML solution and has thread on the forums about it) but because of a few issues, you threw everything out.

You threw away the investment you already had in the integrations that had been done. I don't know if this applies in your case but I've seen where this is typically justified by an environment that is unstable (from sloppy development practices) and the claim that the only way to recover is to completely rewrite it. When in fact, it's highly likely that the existing integrations could've been salvaged and reworked to provide the stability items you mentioned.

You could have implemented a framework with the wM tools you had and moved integrations to that. But the "do it in Java" mentality took over and all that was lost. You decided to use other tools, not wM tools, but then stayed with IS anyway. Again, nothing wrong with a Java approach, but my point is just go the rest of the way. IS seems to provide very little value in your environment. Move the BPM to its own IS server and move the rest to a Java app server environment. You'll be much happier, IMO.

Anyone else have thoughts on these topics? Come join the debate!

[pmarsteau]

What was probably not clear in my recent posts was that we did not thrown anything away from the legacy code. The custom packages coded in the past are still in production, even if they do not make adequate use of the OS resource and badly affect the overall system performance.

What we did was to build up a framework, like a second Java layer on top of IS Java layer itself, to support future customers needs. This increased the scalability of our systems by allowing to run more customers integrations on a single integration line.

Legacy is one very crucial aspect. In our (SaaS) space, where several hundreds of different customers are implemented by our professional services, you cannot throw away any existing code to/or rebuilt it from scratch. Migration strategies are evaluated but someone has to pay the price for it. To leverage the implementation and migration cost, we intend to use a single code line -leveraging the IS features available-, which can be configured by descriptive files. Because we have to support the legacy environment and packages, decisions were taken to not switch to alternate or hybrid integration systems (one which would host "pure" Java apps and the other hosting legacy WM code).

We encapsulated the wM APIs we do need for long the system is hosted on IS in custom Java libraries, so that only these libraries will have to be recoded once a day we would change for another "pure" Java app server environment, if ever. However, IS indeed comes with a lot of features that would be needed to recode if you were starting all over again. Thread management, JDBC Pools, Service invocations framework, scheduled tasks, Monitoring including resubmission-ability, are few of the features we inherit by using the IS as Java container. So no we would not be able to switch without single line of code modification to another system, but this would be feasible by reimplementing the API wrappers. But the added value the framework proven to provide (in term of quality, performance, etc.) in itself justifies the choice of this architecture. And we strongly believe that coding it in Java would never be as efficient and resource saver as if we had coded it in Flow. Is there any drawbacks in terms of readability, coder environment setup, etc, in this solution: yes. Do these overweight the added performance that Java provides over Flow: definitively not.

BTW, I might have induced we stopped using the TN because of the missing protocols. That is not the case. We stopped using it after involving webMethods PS, who themselves recognized the tool was mainly though for B2B purposes, and was not suited for our needs. The IS is mainly used as a (functional) Broker between the customer systems and our internal hosted customers applications (one to many application for one to many customers). So a MoM approach seemed to be more adapted for this kind of system compared to the P2P approach that uses TN. The fact that TN did not support GnuPG or SAML out of the box were just other factors. And initially, we did implement custom TN delivery service to work around this issues.

Note that the statements in my messages only reflects my opinion solely based on the experience I had in our company's environment. No doubt that experiencing other companies environments and architectures running webMethods would probably help me better understand the value Flow provides over Java.

[reamon]

Originally Posted by pmarsteau

What was probably not clear in my recent posts was that we did not thrown anything away from the legacy code. The custom packages coded in the past are still in production, even if they do not make adequate use of the OS resource and badly affect the overall system performance.
...
...Migration strategies are evaluated but someone has to pay the price for it. ...

Ah, I see. Excellent points. Bummer that you can't at least go fix the old ones.

Originally Posted by pmarsteau

...However, IS indeed comes with a lot of features that would be needed to recode if you were starting all over again. Thread management, JDBC Pools, Service invocations framework, scheduled tasks, Monitoring including resubmission-ability, are few of the features we inherit by using the IS as Java container. So no we would not be able to switch without single line of code modification to another system, but this would be feasible by reimplementing the API wrappers.

I think that may be a bit optimistic, but you're the one in the trenches and fully understand the environment so I'll trust that a cutover to use other tools is doable by your team.

I wonder how many of those you'd really need to code yourself. Most of those things are provided out of the box by app servers. They provide bean and thread management, JDBC pooling, services access, etc. Scheduled tasks could be externalized using a job manager like AutoSys.

Originally Posted by pmarsteau

But the added value the framework proven to provide (in term of quality, performance, etc.) in itself justifies the choice of this architecture. And we strongly believe that coding it in Java would never be as efficient and resource saver as if we had coded it in Flow.

This is the root of our difference of opinion. A processing framework is definitely a plus--I'm a big believer in them. I don't claim that Java isn't more efficient in terms of performance but I contend that for the vast majority of integrations and installations, the performance difference is meaningless. And where speed is important (determined through profiling) then one optimizes the key pieces rather than do everything in Java.

Originally Posted by pmarsteau

Is there any drawbacks in terms of readability, coder environment setup, etc, in this solution: yes. Do these overweight the added performance that Java provides over Flow: definitively not.

In the majority of places I've worked, the reverse holds true. Performance is important, but not the most important. Generally speaking, "just enough" performance is sufficient and one doesn't need to resort to dropping to Java for everything.

Originally Posted by pmarsteau

...We stopped using it after involving webMethods PS, who themselves recognized the tool was mainly though for B2B purposes, and was not suited for our needs. The IS is mainly used as a (functional) Broker between the customer systems and our internal hosted customers applications (one to many application for one to many customers). So a MoM approach seemed to be more adapted for this kind of system compared to the P2P approach that uses TN....

I've had debates with many people, including some from PS (and on these forums), about the suitability of TN as a general purpose "broker." I think people lock themselves into thinking that TN is only good for talking with the outside world. I think it applies more broadly than that. Generally speaking, communicating with an outside app can be almost exactly the same as talking with an internal app. You need to configure communication. You need to track things. You need to retry. With message queuing, you end up building such facilities. (Don't get me started on pub/sub. )

IMO, TN is applicable in far more scenarios than people usually think. I've seen home-grown solutions for "internal" interactions that conceptually look an awful lot like TN.

Originally Posted by pmarsteau

...help me better understand the value Flow provides over Java.

I don't know if there is much more than "one doesn't need to be a Java programmer." As you pointed out, there are some things that are easier to code by typing Java source than click and dragging. There some things that are easier to do using Developer and FLOW.

I like the higher level of abstraction that a services approach offers. In Java, one tends to get bogged down in classes and methods, which are generally lower-level constructs. Certainly, higher-level abstractions are doable in Java, coding in Developer just makes you think about things as services more naturally.