Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,107 questions
Get jsonlines response from Rest API and load into SQL DB using ADF pipeline
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 45%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
SarmisthaS
0
Reputation points
Hi Guys,
Requirement is to call Rest APIs which will response data in jsonlines format (not Json array) and load data into table. Tried with Web activity and Set Variable to capture the response but not getting the response in expected format. Can you please suggest the best approach to read jsonline data from Rest API and load into table ?
Output - {
"name": "output",
"value": "\u001f \b\u0000\u0000\u0000\u0000\u0000\u0000\u0003 ]Ys H ~ _Q i;B \u0000 \u001b%YG[ 4\"mMO K , \u0002Ph\u0014 1 \u0011 7 / ̬ A\u0012\u0014 힝 \u001d\u0013Hԙ ] \u00079 PoT N 8\u0012<\u0016 ^ ux\u0002/ <t C \u001d n ٭w* \u001d(}5 SY$ \u000e\u001d \u001c n : u W-\" = \u000f PD R \u0001 '* ʉ\bbɽ\u000f ) ǽh< C7 \u0012x 坘 H\u0004cx \u0001zp Ђ f_\u0003\u0019_s\u001f G< \t_% @G< * 9 %Nڭ \u0006< Ԙ ãk\u0015 \u0017\u001a\u0006 % G ŐR\u0006c\u001a ճ \u0018.C\u001cW_\u00060 a j \u001eE\u0004 ח \u0011;d}5 a ^m[ hRx{3\u001e'Q ۱ m \u0017\u0005 ;\u0011 qK U΅ \u000b + \u0018* q$ \u000f0, [i3 8 bϥM=Eu*b.=\r \u0014 ` \u001e \\ = z}v \u0016R ;1 G\"b { -$l T / b ⹀ \t\u000fX \u000fb d &j\u0011 G P f\u000b!g \u001d `\u0002 \u001cH S_N&Ч \u000b\u0011C7 \u0014 #\u001e1$ F =\u0015v\t x\u0013 \u0016 ë \f)\u000f`a y _\u0000O@ )$ c1 B6 G\u0007l\" xΣ \u001d \u0007 gc`eh\u000f\u0016 ) Y\r\u0007\u0013 \u0013 c D1\u001d\nN d ù \u001a<\u0004 \u001e wp ^ \rG\u000f O U$t\bGF <q # e ݳ \u0001 c\u0011L $; &. 2= \u000b\u0018 b}& ח\u0001 [x4 \u0007N \u0015 / #`+ \u001d 3T \u0012<\u001c \u0011 \u0015 ɿ\u000f 'Ǣ >\u001d 6RZ \u0013 = \u0002 ڜ\u0000 ,; 7 3\u0005 e \u0013 y P \u001e\u0001H h \u0003 \u0015\f\b` vy\u0007 ^]\u0006 D ћf \b B P \b ي ӈ\u0006 6; v L6T p \u000f \r\u0004| jAf lh\u001f: 6: j5% f !J \u0006 F ݫ$F ِ\u0018C鍒h `?ai dE pWe d > ;ذ a[ \u0007 S 5,7 և\u0004 ; d\u001eIh5\u0004vc y\u0000 \\ 8 9\u0004 \u0011\"`S \u0010\f / z \u000b\u001c+#6 f2\u0000 a\u001c6 \u0000n \u0011 X 0 \u0003$ \u0014'\u0013%\u0001S )4 \t 8Sؚ\n, \" k\u00072=\t?ς - 1 SK!J Z \tl LE\u0004 \"\b8 5 + %\u000fh 7 \u001dw 3R 2@a 灬 & \u0001 @ r\f e 6 e 6+\u001cRY\u001f 9<\u000e\u000b E d \n\u000e\u001a 1 ^q J \nG[ \u0012 \nN \" | \u0012 @UL\u0019^` \u001b (A kl \u0000\nT y\u0014\u0001*1 \u0010 \u0001\u001b > Ĉ \f \u0016ڈyj \nZGv \u0018I 0<\u0003?b \u0010N! \u001d4 z \u0010 t 4 # &^ Tk - M\tH H 3iM p\u0016 \tm\u001a *\\\u0006| \u0007Pj@{ fH@ {\u0002ۖSO ަ[ ̓ r\u0011 \u0001 X ] \u0014\u0015 F1߬ \u001d}qm_/ j\r u \t L \nrg趺5 [ g \u0005 z3 e \u0014|QOu H \u0015 z\u0016uj XKd/ \u00120{\u0016 a . n G 1 a & B0 f %\u0004 % O\u000027\u001a$m \u0002 z iԬS \u00004 _tf Db \u001a3_ 6X ۀ 0^ \u0005 w\u0006E V!6 t !\b\n \u0012\u00045\u0011 ª3 h \u0012\f\u0003?= \b b \u001e\"\u0010х \r s V䴌\u0003\u0001\n ^]`h &> \u000f G m ,, q F ˠ 0@ CYD\n v\u000eJIL0 \u0001Yd $\u000b+] \u0011\u0002 R\u000ff j + \u0006Y6 \u0003hA'^ e C z \u0004RL \u0012 F \u0007 O ǰ $aF\u0011 1 \u0005+ Q\f m \b i\u0011 8 I K B % 3 m? 4x Zd \u0011\u0018a@&CJ\u001f\nl C\u0003Lu? / \u0013 fzˁ\u0017\u0000 79 \u0007G \u001f \u0017 `8 +\tL @ d 9 \u0018L\f \u001dP\u000f& \u0007 O\u0010. A {9\u0001. ~ G -j w \u001c_,& v U n\u0013 \u001d / ` 5 }]|N A.> u z\u001f ~ \u0011_ @ ~> I{붻 \u000e +, f _A\u0007 &4C+4 & [h Q q ' Lc K b bE\u0019j 1\u0010r\u0012m\b \u0016\u000e\u000e\u0018 \u0005D% \u0017 \u0018 \u0005 \u001bV! D:A}z O *\u0001\u0015 <r1 L 6 \u0018 - \u0015 \u0018 o F O OS\u000f ӯ N Qs\u001c y U N G\u001a HV L0z G`L\u0011 \u000b Yf][7 nqA \"\u0014 \u0019 b. ɺ\u000bQص v,\f\u0001\u0007 }0& (L2 \u0017 M !\u0005\u001a9gm\u001e < [\u0005Q Hآ( \u001d/J!| } \r5\u001c Vڏ B\u0005 \u0002V s / 9 e\fƿ K / Z V mo\u0013g\r QN\u000b P V_ [ }' f⌥ \bőf HN % ӄώO 2 \u001b $\u0016 s =- V ni F K \u001e \u000e: 9Z\u000e\u0018j2f\u0007b\u001dh\fm\u00047 \\ \u000e \u0011 \u001c \\ \u0011 \u001a1` `\t (DTdžZ B h/\u0019 \u001eE* \u0000G @ nȠ h ,\u001f\u0010y~@n \u000e~ \t ' @ H\u001e\u00014\u0014 / 5\u0013\u0001ΓϽm\u0000 C\"L}\t { l t \u0010]uj v Q\n \r 6 . \ttk \u001ds \u0015 \\$ rpn \u00190 C8o\u0005Me p- q < '\u0011 vpޑA \u001eQ st \u000fb s \r u `5 V&V ƶ \u001dh I \u0005:u\"&he 8 O\u0004w KQ \u001e} s x YYz B +t̥ \u0004hf m \n \t \u0001 \u0010ku\u0006mF . q M \u0001j ܯ _ \u0018 \u001a\u000e N~ x 0H M/\u0018\u0003\u0011\"W q}Ah X5F\u0011 % \u0019X\u0000\u001c jAg 'bMʽ 6\u001b-1>C \u0013: ʗ - Bj) ީ\n X \tGV _Y6J \b`[8 1 ) \\Y\u001a : \n 8 \u0015 %\u0006 \u0005 ɼ 2QKitc\u0010Z ƃ = \"g \u00121 %\u001e \u000e\rZ\rzZqk Gg l F1 \u0014 +rE L Sކ\u0019i X =\u0019 6 C'A \u0003\u0015 s\u0016 E] y (- \u0007\u00126@ D\u0005\u0002{ ֏ \u001f p~1 \u0014 # H! \u0002$ \"ik[h \\l7L\f 0 B \u001f M < \r .\u0010W0 \u001c KK kO\u0016 >ch : 1\u0001\u0006\u0017\b B\u0006 @\u001b\u000f \u0015 R HGC\u0010 %\u000e\u0000Oz6 9 | C\u001e, [ 3 An N \u001d돯 | M \u001e > 8R \u0004 -\b 5e\u0007 0 v\u0016 mC\u00197¢ S\u0001 ar\u0002} )#\u000e0 \t \u0003h 6 $R\u0019 cF \u000eI3 \u0019 \u0002Q # d' h\u0019( ɚ&o \"VVPɧ .fΝ\u0018L\"e 1* N\u0011C ' 5 wT z3PQ0-E'a A N =F ( n 4ۥ : wuJ!r K! nk[\nQ \u001f v \u0014 +o_ FT \u0016 ~ ` \u0016;\u0016\u0011 r U%\u0011 \u00036 \u001d ^ \\;| n=[ V m v\u000b y\u0018oK\u0014 XN\" \u0002#z\u000e ʦx\u0012 T \u0001R 2\u0004˗ } ;1 (0p!n= ]\u001f t0 sd\u0001 \n _\u00152 ̀ \u001f \u0007:e\u0004\r`\u0014 \fBtC<`\u0010 @ y \u0010 f`V%\u001e4# q 鳞:r2\u0017 \u0007 p Gh\u0017 8) 4\u0006Rh \u0002 X\u0004F L\u0018 j Sy=x Z j - b \u0017 \b m#gd \u0015 N ٳ'\u0018 \u0000 T r z \u001dcc t* rw9\u0015w Ț \"\u0000 / *\b \u0013 7| \u001e 5 H\u0018e Y\r8 \u0011 \u000eG Ġ n 5 ( QPAۅ_cJ BŚ p I C / LII#H Q < % M ( \u0011h(R\u0014b\u0001 * 1CoTH 0z 5\u0004h\\ % R\t n3 $\u001awg:5ʻ3Cd d ᵐ\u001e\u001b hI \u0015Rh /w եU < D G \u0017 5D L l \u0010\u0007 ) `SEPo (8 \"̒c v ^ g B + t /5 \u000f |\u0003XJ \u0013 P 5\u0000 ` _ 1+ ! @\u0014 sa \u0001 G_ \u0003 6R}٤ܻC ֭ Jj N it \u001f r n r~ Y u6 -n -\u0014 \t -, \u0002i 4 \u0007zE kk mb3 \u001e c )\r \f EX a\u0006 \n\n\u0015< u,1 T\b H2Fm'ϼ7\t >U! . \bo ZN\u0002+e2 ɻ\u0012 ƓB T b 13 _`r <x`\u001f a \u001dù T 9d k o M \\ _ \r L X r na -( & \u0012 \" \t\fZ A [e m 7 I G o { \u00013{ 3v α dz 'v N z\u0015 1 \u0004| gw\u0006 _ \u0015;á~\u001c\\| 'ø |< $ \u001f X Sb'/ >(x\u0002 gI.b*v@9IJ ,B\u0017 ͗4 9y\u0003q Fb : X p \u0007_\u0000K ;L+ [ Y p qޔ _ \u000b gg< C6\u0010 =Q / 1@*L\u0004 Q>3\u0004 S v ^; g L5+ D@ \u000b4 Fh' b = fV\u000f /\u000e \u0004 8 DV aԤX 1 ٣ r A B B\u000b p \u0017*z \u0007U \u0013 \u001d \u0002Yp3 z \u0004 F <ױ~ V \u0016Ò\u0000 . 7 * |~ /饖\u001a\u001c % Yy V f w e n4v z\\ Dj b$I M \u000ea i IU0͒\u0000lS \u00036a \r%e \u0016iQ\u0015 (Z \u000b-H \"3 \f \u0018\u0004)Aɩm\u0004\u0005\u0004; j\u0012 \u0016m oz > p\u0014a -w2Ȳ0B\t /\u0016\u001b_ (2 \u0019 j l \n;\u00168+k 0 \n\u0014 O \n L h j g m\f\nY >C W)D\u0004F . \u000e< L\u0001b \u001fRLl\u0018ߜ P \u0000 ܒ EAyü } \u000e #PJܛL` K \u001a ȫ 0 z Po6Z fy X \u0016\u00189 n\r μwΒ l Ž \u001d y; S r nsWr\n <'n\u0013 ^]\n ګ\u0014l *\u0017\u0018f'Tdҝ5 \u001e\" {\"\u000b 4\u0000!)\t R\u0017 :X \u0011 46Q 0$1\u0002 r \u000e\u0014 \u0002 \u0016 <\u0018 3 OX͉\u001e7 \u0006\fW\u001ei 0) d\u0015 7bD\u0014 ⱑ \u000et@ 1 4\u001b (Ƙ]`rWFd ڒ+\u0019fUg ^Ug+a f \u0011 w S O\u000b\t 4\u0002M ` {v* \u001f.A~T y F-8 \u0000` ( ^ G 9wX\f B%9% 0\u0005 [ \u0004H\u000f \u001b\b ll \b q\r}h Z \u0014 6 0 \u0013 1 L( \u0013 N(46 \u0002-\u001f D + I > )0<) c t Q H xއ%\u0018Ok\u000f\u0002 e\u001c\f·n Ɏ{ W ۆ U z7 c Q~ *Gy tM ?_ \u0010 o 1 5 \u0010s8 b \u0006@ \u0001 \u0002 K4 \u0003 ~\u0018ư\u001ff ! : \u0007̜c \"M\u0013\u0002 >%;>m د_ +XW*\b T4 \b FoӤ>\f ^ |NQ : Ρ \u0000 ) C{:M n \u0016)1 $. o\u001b\u001b D \u0013 F7 \u0000@ \u00043PJ \u0014\u0007V b!r О ? H o 6 \\ \u000e \u0012 \u000e k 5 /`K6爣\u0013*\u001fFd 3 \u000e EB}\u0007q\r c\" < \u001d}±l 8 J Q \u000e Q#5 a 6/' \\@ \u0004`\u0011 J 7 >\u000f ) \"F Iw EL \u0001 \u0001 u\u001aDI \t\u0003՜ h\u0002 sѻ 9E \u001et V V \u000eR^ fU e\u001a \n ) K>4 \u0017\n np \u00143A3 b#5Y /\u001f = \u0003^ $ My ] \u000f? QHfV l \noVDp -Ӝ /ќ2 R \u0000I q /넗 ( 15 | \\ g 1EN+T 2\u0012E V\u000f 'B\u0011 Q =:>: |tv rut}t{ ϛ\u001a uhlQ \n \u0012j( x ) ,t YE 1 ; 4{ Q \u0017g \u001cpjG b` \u0012k\u0014 z )\u000fҮ \u0015Vw ; \u0014*\u0010 { 6 3y \u0017\u0002 i z j b n : U -\u0014%I f\u0005 nm wm sQ , 6(\t^ g \f jĥ U \u001b \t \u000b\u0011e$ \n | + he . \u0015,W* \u0003Ve 2\u0019 ^ 0\u0005 8k\u0013 \u0017K ] \u0017\u0017 \b1$8 \u0005ƽL A\u001a \tлj (J p r )LC - \u000e V. \u0017 t\u0015V $ K \u0019݅ q \u0002 MHs K \u0014~۾\u0019p cƫ`b\u0013 N B˿\u0015U j/\u0017t _\u0010W %rH\u00153ȕ'\n \u0016TT\u0006 \u00028 @x\fd Sg & 9 S 'rK3 e GZE# J '$ e~ Ʌ p[ \u001aQ\u0007 D0\u000e, b \b [ ,g \u0015Jt [8T W \u0014hN] x \u0000 ȣ\f4 % \u0002 P<\u0004\u0003 \u0011 $ 3Y\u0011 \u001fe \u0002\u001f\u0006 =k<!ʂ q+ 4 \u0005 ;v \u0014 \u0012\u000b l݄ S.\u0005\u001b n \nA \u0010 C k F \u0016 I l ^ {^(\u001fD d u\u000e ͚P 6 f W \u0012 \u0001 ( & X UX.% EA WU ]u >\u0019w ` r\u0013 ą ~ G J D\u000f &\u0013~\u00177 \t' \u001d \u000e' &U \tS_|j һ5\u0016& Tڌ\u0018 ® kK\u0006 rd D 8 W< \u001b өvڥ cĵơ uw$ v@ ^ ] @\\\u0011P\u001c*Xi \u0003^ $ \n6= W*@ D~ \u0010y+0\u0017\u001d \u0016 U4\u0004 y8$ \" 4%fB\u0003k 0\u0015v ȴ,l /K a \u0017 JL\rp F`0 Ÿt U O 丧\u0019 X\u0001Ӓj\u0017 V p \nf & \rc\"ä< \u000e L ( \u0006 rD \u001eˮ ul _P \r \u0016 ۛ+ ,K 5 t U Wui |J+ ί E~ 7 19= _ \u0019 \u0019 ĺ>bw-1[ \b f y Y 5j ! !Q rk`I: A a ٣б 鸨(\u0017 XW o g y\" I l \u001e%g C\u0014 \fЈL ! >0}\u0018#ٗԳ9 Ц>: V l 4\u0004N Z 1 Q\u0003\u0002 {\u0010k7BW\u0018{ L \u001d W \u0017 a r \u0006Be ' \u001a [ /nz N >\\ k\u0006 `], y W f\u0011 \u001c \u0007 O<\u000fv}k1 `Γ ݯ Z lH= PX7 \n6% .3\u0007Oz, J7l Q\u0006 \u0012 k#P\"t< \u000f \u0007 ; \u0013n a s:2 ! T 星2 \u001bx}4~W<Q\u000b S \u000eh\u0006 G '֞2 )& l * ldt|qva B`4 t`+\u0013\u0004\u0003\nxu _ an\t ӵ 2 8 \u0019\f\u001b ]\u001c br0t F \u0016 Y B,ʆ ) A/ L\u0014d0\u001a}\b\u001f\u00187 :^ \u001e<Jo \u001c 4 \u0014 \u0010\u001c\u001aL u ܑs \u001es\u0019a \r \rL z UuX~ W Z \u000f\u0005 y \u0003D j \u0019> B \t c |\u0011q 9 p r P d:史 ` Z q \u0018 dzy } Z-7 \u0006I% \u001fT[ 5ӛ 7o* 6 Nڌo U f n ; ڣh:E ~\u001e \u0018 \" !&m ; &U G\u001f\u0007gN \u0013 \\C \u00012 \u000b (`V ˂ \u0016& ] \u0015 Ĕ\fӟ\u000f1 \u0014 鏏 [ i|\u001e }J 7 k \f G\tޗ\u0005h .ذ\u0002 [nrF P aCv:8+c R e V ]\u001a! Z \u0017 9 X : X\u0015 BJ t 6 4 6\u0012 \u0002 \u0002 ;\u0006 \u0003\u0001 Y σ j j \u001c 5\u0001+5 ?@@e5\u0000opgo b O 6Z ƞb i a ? ܽ y ŹY\b vZmRձh\u0000\u000b ֯ Ms A \u0017 3J \u0002\u0003 / 4\u0007P \u000e[* \u0001 \t \u001a \u0018k$ɟ m ـ \u0006 ߬ f o{ \u001c ҫ d : ^y B \u0001W` e w\u0000\u0000"
}
Web Activity - {
"name": "Web1",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"method": "GET",
"headers": {
"x-api-key": "ghghjkjkh",
"Accept": "application/jsonlines",
"Content-type": "application/x-www-form-urlencoded"
},
"url": "<url>"
}
}
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth 12,900 Reputation points Microsoft Vendor2024-04-15T16:59:51.8733333+00:00 Thanks for reaching out to Microsoft Q&A
It seems like you’re trying to consume a REST API that returns data in the JSON Lines format (jsonl) and load this data into a table. JSON Lines is a convenient format for storing structured data that may be processed one record at a time. It works well with Unix-style text processing tools and shell pipelines.
Here’s a general approach to solve your problem:
- Call the REST API: Use an HTTP client library in your programming language of choice to send a GET request to the API.
Read the response: The API response will be a string where each line is a separate JSON object. You can read this response line by line. Each line in the response is a stringified JSON object.
Parse each line: For each line, use a JSON parsing library to convert the stringified JSON object into a data structure that your programming language can work with.
Load into a table: Once you have the data in a structured format, you can load it into a table. The specifics of this step will depend on what kind of table you’re working with (e.g., a SQL table, a Pandas DataFrame, etc.).
Here’s a Python example that uses the
requestsandjsonlibraries to call a REST API and parse the response:import requests import json # Send a GET request to the API response = requests.get('http://api.example.com/data') # Split the response text into lines lines = response.text.splitlines() # Parse each line as JSON and print it for line in lines: data = json.loads(line) print(data)In this example,
datais a Python dictionary that you can work with. You could add code to load this data into a table.Please replace
'http://api.example.com/data'with your actual API endpoint.Remember to handle exceptions and edge cases as necessary for your specific use case. This might include handling network errors, dealing with rate limits, and managing large responses.
As for the output you’ve shared, it seems to be in a binary or encoded format. If this is the output from the API, you might need to check if the API is supposed to return data in this format or if there’s an issue with how the API is being called. If the API is supposed to return JSON Lines data, the output should be readable text, not binary or encoded data. If the API is returning binary data, you might need to decode it before you can process it. The specifics of how to do this will depend on the encoding used by the API. If you’re unsure, you might need to check the API’s documentation or contact the API provider for support.
Hope this helps. Do let us know if you any further queries.
-
SarmisthaS 0 Reputation points
2024-04-16T08:24:48.1833333+00:00 Thanks for your reply. But is there any way to read the API response as JSON Line format in ADF activities (Copy activity or Web activity or any other) ?
The output of API is not expected to be in binary or encoded data. From Postman , I am getting the JSON line response as readable text but when calling it from ADF using WEB Activity (setup as mentioned in question) getting in that format. Any setup is required in HEADER to call the API in Web Activity ?
-
phemanth 12,900 Reputation points Microsoft Vendor
2024-04-16T09:08:58.43+00:00 @SarmisthaS In Azure Data Factory (ADF), you can use the Web Activity to call a REST API and handle the response. However, ADF’s built-in activities like Web Activity or Copy Activity may not natively support JSON Lines format.
Here’s a workaround you could consider:
- Lookup Activity: Use the Lookup activity to read lines from the API response.
- ForEach Activity: Add a ForEach activity to iterate through each line. Inside the ForEach loop, you can parse each line as a JSON object and process it as needed.
Regarding the headers in the Web Activity, you can set them in the
headersproperty of the activity. Here’s an example of how to set theContent-Typeheader:"headers": { "Content-Type": "application/json" }Please replace
"application/json"with the actual content type required by your API. -
phemanth 12,900 Reputation points Microsoft Vendor
2024-04-17T08:35:19.7833333+00:00 @SarmisthaS We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 35%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sarmistha Sarkar 0 Reputation points2024-06-12T12:13:52.0133333+00:00 HI Phemanth,
I used Databricks to read the jsonlines from Open API using Python script. It can read and keep the data into a file in Azure blob storage. but when I am executing that notebook from ADF pipeline, it is failing with error - 'module not found': azure
if I include the azure module installation step in the notebook then it is executing from ADF pipeline. Can you please tell me how I can avoid to installation step in my notebook script to run every time.
below is my script in Databricks notebook in python -
===========================
import requests
%pip install azure-storage-blob ///want to avoid to execute that installation in script each time of execution
dbutils.library.restartPython()
from azure.storage.blob import BlobServiceClient //getting error for that azure module
#Make the API call to get the data in jsonlines format
response = requests.get('XXXXXXXX'),
headers={'X-API-Key': 'XXXXXXX','Content-Type': 'application/jsonlines'})
#Get the data from the response
data = response.text
#Create a BlobServiceClient to connect to Azure Blob Storage
blob_service_client = BlobServiceClient.from_connection_string('DefaultEndpointsProtocol=https;AccountName=<XXX>;AccountKey=<XXXX>;EndpointSuffix=core.windows.net')
#Check if the container exists
container_name = 'test'
container_client = blob_service_client.get_container_client(container_name)
#If the container does not exist, create it
if not container_client.exists():
container_client.create_container()
#Create a blob client to upload the data
blob_client = container_client.get_blob_client('generalriskiq')
#Upload the data to the blob
blob_client.upload_blob(data, overwrite=True)
-
phemanth 12,900 Reputation points Microsoft Vendor
2024-06-14T08:45:01.2766667+00:00 @SarmisthaS The scope of your question is different compared to the original scope of the question asked.
I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.
Sign in to comment
Sign in to answer