Decoding a URL Encoded POST String

Dedicated to DataStage and DataStage TX editions featuring IBM<sup>®</sup> Service-Oriented Architectures.

Moderators: chulett, rschirm

Post Reply
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Decoding a URL Encoded POST String

Post by pbttbis »

Hi,

Is there any functionality in Datastage to allow me to decode an url encoded post string such as the below:

calcData=%3cMotorInsuranceRequest+xmlns%3d%22http%3a%2f%2fwww.insurethebox.com%2fMotorInsuranceRequest.xsd%22+xmlns%3axsi%3d%22http%3a%2f%2fwww.w3.org%2f2001%2fXMLSchema-instance%22+xmlns%3axsd%3d%22http%3a%2f%2fwww.w3.org%2f2001%2fXMLSchema%22+xmlns%3aemp%3d%22http%3a%2f%2fwww.insurethebox.com%2fEmployment.xsd%22+xmlns%3adrv%3d%22http%3a%2f%2fwww.insurethebox.com%2fDriver.xsd%22+xmlns%3aclm%3d%22http%3a%2f%2fwww.insurethebox.com%2fClaims.xsd%22+xmlns%3aconv%3d%22http%3a%2f%2fwww.insurethebox.com%2fConvictions.xsd%22+xmlns%3amad%3d%22http%3a%2f%2fwww.insurethebox.com%2fMainAdditionalDriver.xsd%22+xmlns%3aveh%3d%22http%3a%2f%2fwww.insurethebox.com%2fVehicleTypes.xsd%22+xmlns%3apol%3d%22http%3a%2f%2fwww.insurethebox.com%2fPolicy.xsd%22%3e%0d%0a++%3cProposer%3e%0d%0a++++%3cemp%3aEmploymentTitle%3e824%3c%2femp%3aEmploymentTitle%3e%0d%0a++++%3cdrv%3aHasClaim%3eNo%3c%2fdrv%3aHasClaim%3e%0d%0a++++%3cdrv%3aHasConviction%3eNo%3c%2fdrv%3aHasConviction%3e%0d%0a++++%3cdrv%3aGender%3eM%3c%2fdrv%3aGender%3e%0d%0a++++%3cdrv%3aMaritalStatus%3e1%3c%2fdrv%3aMaritalStatus%3e%0d%0a++++%3cdrv%3aLicenceType%3e1%3c%2fdrv%3aLicenceType%3e%0d%0a++++%3cdrv%3aDrivingQualifications%3e3%3c%2fdrv%3aDrivingQualifications%3e%0d%0a++++%3cdrv%3aTitle%3eMr%3c%2fdrv%3aTitle%3e%0d%0a++++%3cdrv%3aForeName%3eShaun%3c%2fdrv%3aForeName%3e%0d%0a++++%3cdrv%3aSurname%3etest%3c%2fdrv%3aSurname%3e%0d%0a++++%3cdrv%3aDateOfBirth%3e02%2f04%2f1993%3c%2fdrv%3aDateOfBirth%3e%0d%0a++++%3cdrv%3aIsMainDriver%3etrue%3c%2fdrv%3aIsMainDriver%3e%0d%0a++++%3cdrv%3aLicenceHeldSinceMonths%3e01%3c%2fdrv%3aLicenceHeldSinceMonths%3e%0d%0a++++%3cdrv%3aLicenceHeldSinceYears%3e2012%3c%2fdrv%3aLicenceHeldSinceYears%3e%0d%0a++++%3cdrv%3aMedicalConditions%3e0%3c%2fdrv%3aMedicalConditions%3e%0d%0a++++%3cdrv%3aDriveOtherVehicles%3eNo%3c%2fdrv%3aDriveOtherVehicles%3e%0d%0a++++%3cdrv%3aResidentSinceMonths%3e04%3c%2fdrv%3aResidentSinceMonths%3e%0d%0a++++%3cdrv%3aResidentSinceYears%3e1993%3c%2fdrv%3aResidentSinceYears%3e%0d%0a++++%3cdrv%3aHasCriminalConviction%3eNo%3c%2fdrv%3aHasCriminalConviction%3e%0d%0a++++%3cmad%3aAddress%3e%0d%0a++++++%3cmad%3aHouseNumberName%3e57%3c%2fmad%3aHouseNumberName%3e%0d%0a++++++%3cmad%3aAddressLineOne%3eWhite+Swan+Close%3c%2fmad%3aAddressLineOne%3e%0d%0a++++++%3cmad%3aAddressLineTwo%3eKillingworth%3c%2fmad%3aAddressLineTwo%3e%0d%0a++++++%3cmad%3aAddressLineThree%3eNewcastle+upon+Tyne%3c%2fmad%3aAddressLineThree%3e%0d%0a++++++%3cmad%3aAddressLineFour%3e%3c%2fmad%3aAddressLineFour%3e%0d%0a++++++%3cmad%3aPostalCode%3eNE12+6UG%3c%2fmad%3aPostalCode%3e%0d%0a++++%3c%2fmad%3aAddress%3e%0d%0a++++%3cmad%3aMobilePhone%3e07414+545854%3c%2fmad%3aMobilePhone%3e%0d%0a++++%3cmad%3aEmailAddress%3eamc080120161%40bigggg.com%3c%2fmad%3aEmailAddress%3e%0d%0a++++%3cmad%3aPreferredPaymentMethod%3eAnnually%3c%2fmad%3aPreferredPaymentMethod%3e%0d%0a++++%3cmad%3aNoClaimDiscountBonus%3e1%3c%2fmad%3aNoClaimDiscountBonus%3e%0d%0a++++%3cmad%3aDeclinedMotorInsurance%3eNo%3c%2fmad%3aDeclinedMotorInsurance%3e%0d%0a++++%3cmad%3aHomeOwner%3eF%3c%2fmad%3aHomeOwner%3e%0d%0a++%3c%2fProposer%3e%0d%0a++%3cAdditionalDriverCollection+%2f%3e%0d%0a++%3cVehicle%3e%0d%0a++++%3cveh%3aAnnualMileage%3e8000%3c%2fveh%3aAnnualMileage%3e%0d%0a++++%3cveh%3aIsVehicleModified%3eNo%3c%2fveh%3aIsVehicleModified%3e%0d%0a++++%3cveh%3aVehicleNightLocation%3e2%3c%2fveh%3aVehicleNightLocation%3e%0d%0a++++%3cveh%3aVehicleKey%3e17553502%3c%2fveh%3aVehicleKey%3e%0d%0a++++%3cveh%3aVehicleRegistration%3eNH08UHR%3c%2fveh%3aVehicleRegistration%3e%0d%0a++++%3cveh%3aSeats%3e5%3c%2fveh%3aSeats%3e%0d%0a++++%3cveh%3aHasTracker%3eNo%3c%2fveh%3aHasTracker%3e%0d%0a++++%3cveh%3aVehicleDrivingPosition%3e2%3c%2fveh%3aVehicleDrivingPosition%3e%0d%0a++++%3cveh%3aVehicleSecurityDevice%3e5%3c%2fveh%3aVehicleSecurityDevice%3e%0d%0a++++%3cveh%3aYearOfManufacture%3e2008%3c%2fveh%3aYearOfManufacture%3e%0d%0a++++%3cveh%3aVehicleDayLocation%3e7%3c%2fveh%3aVehicleDayLocation%3e%0d%0a++++%3cveh%3aBusinessMileage%3e0%3c%2fveh%3aBusinessMileage%3e%0d%0a++++%3cveh%3aVehicleImported%3eNo%3c%2fveh%3aVehicleImported%3e%0d%0a++++%3cveh%3aVehicleMarketValue%3e4045%3c%2fveh%3aVehicleMarketValue%3e%0d%0a++%3c%2fVehicle%3e%0d%0a++%3cPolicyDetail%3e%0d%0a++++%3cpol%3aIncludeProtectedNoClaimsBonus%3eF%3c%2fpol%3aIncludeProtectedNoClaimsBonus%3e%0d%0a++++%3cpol%3aPolicyStartDate%3e08%2f01%2f2016%3c%2fpol%3aPolicyStartDate%3e%0d%0a++++%3cpol%3aRegisteredKeeper%3e0%3c%2fpol%3aRegisteredKeeper%3e%0d%0a++++%3cpol%3aRegisteredOwner%3e0%3c%2fpol%3aRegisteredOwner%3e%0d%0a++++%3cpol%3aVehiclePurchasedDate%3e01%2f05%2f2015%3c%2fpol%3aVehiclePurchasedDate%3e%0d%0a++++%3cpol%3aPolicyUsage%3e1%3c%2fpol%3aPolicyUsage%3e%0d%0a++++%3cpol%3aCoverLevel%3e01%3c%2fpol%3aCoverLevel%3e%0d%0a++++%3cpol%3aVoluntaryExcess%3e2%3c%2fpol%3aVoluntaryExcess%3e%0d%0a++%3c%2fPolicyDetail%3e%0d%0a%3c%2fMotorInsuranceRequest%3e&calcType=Full+Premium+Calc+TBB&transform=%2fTransforms%2fctmTransformTBB&calcResponse=%2fresponses%2fctmResponseTBB&source=ctm&test=test

Thanks,

Shaun
PBT TBIS Consultant
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

I haven't seen anyone do it, but it shouldn't be that difficult, provided you know all the different "translations" for the various escape sequences that are in here. I am not familiar with this, but clearly it looks like there are escapes for windows style carriage return line feeds [ %0d%0a ] and other such hex character references. I'd probably write a UV/Basic routine with a lot of nested ereplace()'s and use the Basic Transformer, but that's me. ; ) I'm sure there are a lot of ways to do it.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
pbttbis
Premium Member
Premium Member
Posts: 36
Joined: Thu Dec 11, 2014 3:30 am
Location: South Africa
Contact:

Post by pbttbis »

well there are 224 replacement pairs I believe based on http://www.w3schools.com/tags/ref_urlencode.asp

Some options we were considering:

1) Write a routine
2) Use the JAVA stage and make use of String result = java.net.URLDecoder.decode(url, "UTF-8");
3) Perhaps using the transformer looping through the pairs and doing the replacement with ereplace

Currently we have a working job that does the decode using the External Filter step and the following perl script:

perl -pi.back -e 'y/+/ /;s/%([\\da-f]{2})/pack H2,$1/gie;s/[\r\n]//g'
PBT TBIS Consultant
Post Reply